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GAO 

General  Accounting  Office  (before  July  7,  2004);  Government 
Accountability  Office  (after  July  7,  2004) 

ICD 

Initial  Capabilities  Document 

INCOSE 

International  Council  on  Systems  Engineering 

IOC 

Initial  Operational  Capability 

JCIDS 

Joint  Capabilities  Integration  and  Development  System 

KPPs 

Key  Performance  Parameters 

LRIP 

Low-Rate  Initial  Production 

MAIS 

Major  Automated  Information  Systems 

MDA 

Milestone  Decision  Authority 

MDAP 

Major  Defense  Acquisition  Program 

MDD 

Materiel  Development  Decision 

MS 

Milestone 

ORD 

Operational  Requirements  Document 

PAUC 

Program  Acquisition  Unit  Cost 

PDR 

Preliminary  Design  Review 

PPBE 

Planning,  Programming,  Budgeting,  and  Execution 

R&D 

Research  and  Development 

RDT&E 

Research,  Development,  Test,  and  Evaluation 

SAR 

Selected  Acquisition  Report 

SBIRS 

Space-Based  Infrared  System 

SEI 

Software  Engineering  Institute 

SEM 

Structural  Equation  Modeling 

sv 

Schedule  Variance 

TDS 

Technology  Development  Strategy 

TRL 

Technology  Readiness  Level 

USD(AT&L) 

Under  Secretary  of  Defense  for  Acquisition,  Technology,  and  Logistics 

WBS 

Work  Breakdown  Structure 

WSARA 

Weapon  Systems  Acquisition  Reform  Act  of  2009 
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I.  INTRODUCTION 


A.  OVERVIEW 


The  current  policy  climate  in  the  federal  government  is  one  of  increasing  fiscal 
austerity.  The  Department  of  Defense  (DoD)  spent  approximately  $500  billion  on 
contracted  goods  and  services  in  fiscal  year  (FY)  2010,  of  which  $135  billion  was  spent 
on  procurement.  Under  Secretary  of  Defense  for  Acquisition,  Technology,  and  Logistics 
(USD[AT&L])  Ashton  Carter  released  a  memo  in  September  2010  providing  guidance 
for  how  the  DoD  could  achieve  an  estimated  $100  billion  in  management  and  contracting 
efficiencies  over  the  course  of  five  years  (Carter,  2010).  Decreasing  budgets  will  require 
acquisition  professionals  to  do  more  with  less. 

The  Government  Accountability  Office  (GAO,  2011)  reported  the  FY2008 
portfolio  of  96  Major  Defense  Acquisition  Programs  (MDAPs)  collectively  ran  $303 
billion  over  initial  program  budget  and  were  an  average  of  22  months  behind  schedule  in 
delivering  initial  capability.  The  analysis  of  cost  variance  (CV),  schedule  variance  (SV), 
and  engineering  variance  (EV)  differences  across  MDAPs  could  highlight  defense 
acquisition  industry  segments  with  program  management  best  practices.  If  not 
attributable  to  program  management  practices,  differences  across  MDAPs  could  point  to 
differing  levels  of  technology  insertion  risk  or  system  integration  complexity. 

As  acquisition  professionals  have  attempted  to  increase  efficiency  and  examine 
current  contracting  practices,  one  area  of  interest  has  been  contract  type.  There  has  been 
a  new  emphasis  within  the  acquisition  community  to  increase  the  amount  of  fixed-price 
contracts.  At  one  time,  the  DoD  attempted  to  impose  fixed-price  incentive  contracts  on 
efforts  in  which  significant  invention  could  be  anticipated,  although  recently  the  use  of 
cost-plus-award- fee  contracts  has  become  widespread  (Carter,  2010).  Under  Secretary 
Carter  advocated  the  use  of  fixed-price  incentive  firm  target  contracts  in  the  place  of  cost- 
plus-award-fee  contracts  wherever  practicable  (Carter,  2010). 

According  to  Table  6.8  of  the  National  Defense  Budget  Estimates  for  FY2012 
(Under  Secretary  of  Defense  [USD]  Comptroller,  2011),  also  known  as  the  DoD 
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Greenbook,  research  and  development  (R&D)  contracts  for  FY2010  totaled  $80  billion. 
Increasing  the  number  of  R&D  contracts  issued  on  a  fixed-price  incentive  basis  could 
cause  significant  savings  based  on  the  presumption  that  fixed-price  incentive  contracts 
decrease  cost  overruns.  During  the  engineering  and  manufacturing  development  (EMD) 
phase  of  programs,  contracts  funded  through  research,  development,  test,  and  evaluation 
(RDT&E)  are  of  particular  interest  because  they  represent  the  last  stage  in  which  large 
cost-plus  contracts  are  typically  awarded.  However,  restricting  R&D  work  to  a  fixed- 
price  incentive  basis  could  negatively  affect  the  detection  and  solving  of  problems  early 
in  a  program. 

Contracted  companies  may  stop  researching  all  problems  and  alternatives  in  order 
to  increase  their  profit  margin  on  the  fixed-price  incentive  contracts,  provided  that  the 
contract  specification  requirements  are  satisfied.  Properly  constructed  incentives  may 
mitigate  the  risks  of  contractors’  scope  of  research  for  a  materiel  solution,  but 
quantitative  incentives  may  not  be  able  to  motivate  all  the  desired  behaviors. 
Deficiencies  in  the  identification  of  MDAP  problems  could  increase  costs  in  the  long 
term.  For  example,  if  a  contractor  eliminated  tests  during  RDT&E  to  save  money  for  the 
company,  then  it  could  result  in  increased  expenses  to  the  government  to  fix  the 
unidentified  problems  in  the  future. 

1.  Impact  of  Acquisition  Failures  on  the  Warfighter  and  the  Public 

Cost  overruns  and  schedule  delays  have  a  significant  impact  on  the  warfighter  and 
the  taxpayers.  Deputy  Secretary  of  Defense  William  J.  Lynn  III  (2009)  stated, 

American  taxpayers  and  our  men  and  women  in  uniform  are  understandably 
skeptical  when  they  hear  promises  to  reform  the  Defense  Department’s  sprawling 
acquisition  system,  which  often  delivers  major  weapons  systems  to  our  troops 
years  behind  schedule  and  billions  of  dollars  over  budget. 

There  have  been  numerous  studies  that  have  focused  on  improving  defense  acquisitions, 

but  the  DoD  repeatedly  experiences  the  same  problems;  cost  and  schedule  growth  have 

continued  over  the  last  30  years  regardless  of  all  the  acquisition  reforms,  congressional 

studies,  and  DoD  reports  repetitively  highlighting  the  same  issues  (Schwartz,  2010). 

These  include  specific  studies  of  successes  (Dillard  &  Ford,  2009)  and  spectacular 
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failures  (General  Accounting  Office  [GAO],  1992),  as  well  as  more  systemic  reforms 
(Department  of  Defense  Authorization  Act,  1982;  DoD,  2011).  The  aim  of  these  efforts 
was  to  improve  elements  of  the  acquisition  system,  but  they  have  not  effectively 
addressed  the  reinforcing  relationships  present  between  fiscal  constraints,  state-of-the-art 
technical  requirements,  and  the  acute  needs  of  the  government,  as  explored  in  the 
discussion  of  risk  by  acquisition  phase  in  Chapter  II  of  this  research. 

Program  oversight  generally  addresses  specific  problems  and  not  the  process  as  a 
whole.  Accordingly,  it  is  extremely  difficult  to  balance  the  pressures  exerted  by  different 
reporting  requirements  over  a  program’s  life  cycle;  it  is  possible  that  these  reporting 
requirements,  intended  to  improve  acquisition  outcomes,  may  decrease  the  probability  of 
program  success  due  to  their  onerous  nature  (Wood  &  Moseley,  2011).  Asa  report  from 
Business  Executives  for  National  Security  (BENS,  2009)  stated,  “Defense  acquisition 
revolves  around  15-year  programs,  5-year  plans,  3-year  management,  2-year  Congresses, 
18-month  technologies,  1-year  budgets,  and  thousands  of  pages  of  regulations”  (p.  1). 
Despite  heavy  oversight,  the  DoD  continues  to  manage  programs  that  are  delivered  late 
and  that  experience  cost  overruns,  all  while  additional  layers  of  oversight  are  inserted. 

2.  History  of  Acquisition  Failures 

The  current  management  problems  that  acquisition  programs  face  include  the 
failure  to  control  costs  and  schedules.  According  to  Defense  Acquisition  Specialist 
Moshe  Schwartz  (2010),  “More  recently,  concerns  over  defense  acquisitions  have 
centered  around  significant  cost  overruns,  schedule  delays,  and  an  inability  to  get  troops 
in  the  field  the  equipment  they  need  when  they  need  it”  (p.  13),  but  acquisition  and 
contracting  failures  are  a  persistent  problem.  During  the  Civil  War,  President  Abraham 
Lincoln  forced  Secretary  of  War  Simon  Cameron  to  resign  due  to  corruption  and 
mismanagement  of  contracting  in  the  War  Department  (The  Lincoln  Institute,  2011).  The 
calamity  of  acquisition  management  has  existed  in  the  United  States  military  since  the 
Civil  War,  and  as  long  as  there  are  contracts,  contracting  oversight  failures  will  be 
difficult  to  eliminate.  Mitigating  the  cost  and  frequency  of  oversight  failures  through  the 
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sound  use  of  contract-  and  program-level  indicators  of  future  variance  may  offer  some 
relief  from  these  failures. 


3.  Importance  of  Research  and  Development 

R&D  activities  are  necessary  to  develop  many  technologies  used  in  warfighting 
systems.  R&D  directly  contributes  to  program  success  by  demonstrating  successively 
higher  levels  of  integration  and  realism  in  a  system’s  or  subsystem’s  technology.  This 
increasing  level  of  demonstrated  ability  eliminates  some  technical  risk.  R&D  for  some 
systems  and  components  begins  before  Milestone  (MS)  A.  The  R&D  discussions  in  this 
research  focus  on  later  system  R&D,  specifically  during  the  EMD  acquisition  phase. 

4.  Current  Acquisition  Climate 

The  proposed  2010  defense  budget  accounted  for  20%  of  the  federal  budget,  of 
which  one  third  was  for  defense  acquisitions  (BENS,  2009).  In  July  2011,  Congress 
debated  cutting  $866  billion  from  the  defense  budget  over  the  next  ten  years  (Ewing, 
2011).  By  comparison,  the  DoD  requested  a  budget  authority  of  $671  billion  for  FY2012 
(USD  Comptroller,  2011).  The  DoD  spends  approximately  $400  billion  a  year  on 
contracted  goods  and  services  (Carter,  2010).  It  appears  the  DoD  must  continue 
satisfying  more  requirements  with  similar  budgets,  or  as  Under  Secretary  Carter  (2010) 
says,  “do  more  without  more”  (p.  1). 

B.  PURPOSE 

Our  purpose  in  this  research  is  to  identify  the  presence  of  failure  or  success 
indicators  before  the  production  phase  for  DoD  acquisition  programs  using  both  program 
and  contract  variances.  To  identify  the  indicators  influential  in  program  failure  or 
success,  we  examine  the  variances  during  the  production  phase  by  analyzing  statistical 
relationships  of  preproduction  program  and  contract  variances.  We  determined  these 
relationships  by  examining  the  impact  of  cost,  schedule,  and  engineering  variances 
during  research  and  development  to  later  period  variances  of  program  execution. 

CY,  SV,  and  EV  within  individual  programs  have  only  been  qualitatively  linked. 
Under  Secretary  Carter  (2010)  noted, 


ACQUISITION  RESEARCH  PROGRAM 

GRADUATE  SCHOOL  OF  BUSINESS  &  PUBLIC  POLICY 

NAVAL  POSTGRADUATE  SCHOOL 


-4- 


As  all  programs  compete  for  funding,  the  usual  result  is  that  a  program 
settles  into  a  level-of-effort  pattern  of  annual  funding  that  does  not  deviate 
much  from  year  to  year.  The  total  program  cost  is  the  level-of-effort  times 
the  total  length  of  the  program.  Thus  a  one-year  extension  of  a  program 
set  to  complete  in  10  years  can  be  expected  to  result  in  a  10  percent 
growth  in  cost  as  the  team  working  on  the  project  is  kept  on  another  year. 

(P-4) 

Further  understanding  the  interrelationship  of  CV,  SV,  and  EV  could  allow  program 
managers  to  better  understand  the  full  programmatic  impact  of  a  cost,  schedule,  or 
technical  problem  during  EMD.  identifying  outliers  that  defy  the  norm  of  a  positive  CV, 
SV,  and  EV  reinforcing  loop  could  also  aid  future  researchers  in  identifying  best 
practices  in  recovering  from  a  program  cost  overrun,  schedule  slip,  or  technical 
deficiency.  A  reinforcing  loop  exists  when  CV,  SV,  and  EV  interrelate  and  cause  each 
other  to  continuously  rise  in  a  reinforcing,  rather  than  stabilizing,  manner.  The  cost  and 
schedule  variances  that  are  part  of  the  earned  value  contract  reports  in  SARs  may  also  be 
linked  to  the  program-level  variances. 

Performing  a  quantitative  analysis  on  program  and  contract  variances  could 
provide  statistical  predictions  of  future  effects.  The  results  will  help  at  all  levels  of 
program  management,  including  during  the  initial  programming  of  requirements.  The 
results  will  also  show  which  contract  and  program  variances  have  the  greatest  effect, 
thereby  helping  program  managers  to  prioritize  which  variances  to  give  their  finite 
attention.  Beyond  this,  formulating  and  estimating  this  linkage  will  enhance 
understanding  of  programs  and  contracts  and  will  close  gaps  between  the  closely  related 
acquisition  communities  of  program  management  and  contracting. 

C.  RESEARCH  QUESTIONS 

The  problems  with  MDAPs  running  over  budget  and  behind  schedule  gave  rise  to 
several  research  topics  that  must  be  investigated.  Of  particular  interest  to  us  was  the 
increased  use  of  fixed-price  contracts  recommended  by  the  DoD  and  the  effects  of 
contract  variances  on  program  variances.  In  order  to  focus  these  general  research  topics, 
we  proposed  the  following  questions. 
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1.  Primary  Questions 

1 .  What  effect  does  fixed-price  R&D  have  on  production  cost,  schedule,  and 
technical  performance? 

2.  Do  different  segments  of  MDAPs  (e.g.,  fighters,  tanks,  missiles,  satellites) 
exhibit  differing  cost  and  schedule  growth? 

3.  Does  early  CV,  SV,  or  EV  serve  as  a  leading  indicator  of  later-period  CV, 
SV,  and/or  EV  in  either  EMD  or  post  Milestone  C? 

2.  Secondary  Questions 

1 .  What  portion  of  MDAPs  have  fixed-price  incentive  R&D  contracts? 

2.  Is  there  qualitative  information  to  support  the  assertion  that  fixed-price 

contracts  during  the  EMD  phase  hinder  the  identification  of  program  problems? 

3.  If  the  effect  of  fixed-price  R&D  is  measurable,  are  the  variances  larger 
with  regard  to  cost,  schedule,  or  engineering  during  EMD  and  production? 

4.  Based  on  the  results  found  in  this  research,  can  any  definitive  policy 
recommendations  be  made? 


D.  SCOPE  AND  METHODOLOGY 


In  this  research,  we  include  the  following:  (1)  a  review  of  general  acquisition  CV, 
SV,  and  EV  performance;  (2)  an  in-depth  review  of  the  effect  of  fixed-price  R&D  during 
EMD  on  program  CV,  SV,  and  EV;  and  (3)  a  discussion  of  the  interrelationships  of  CV, 
SV,  and  EV.  We  conclude  this  research  report  with  a  recommendation  for  R&D  contract 
type  and  a  discussion  of  the  implications  of  overall  CV,  SV,  and  EV  performance. 


In  this  research,  we  first  examine  the  available  archival  MDAP  data  using 
traditional  descriptive  statistics  in  order  to  determine  whether  identifiable  patterns  exist 
among  contracts  within  MDAPs.  This  analysis  helps  close  existing  gaps  between  our 
understanding  of  program  management  and  contract  management.  In  order  to  perform 
this  analysis,  we  collected  cost,  schedule,  and  engineering  variance  data,  including  post 
MS  C  production  and  deployment  phase  data,  for  3 1  MDAPs  from  Selected  Acquisition 
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Reports  (SARs)  for  multiple  years.  The  database  consists  of  cross-sectional,  time-series 
data.  The  cross-section  includes  multiple  programs  and  their  attendant  contracts  during  a 
particular  year.  The  time-series  consists  of  individual  programs  and  their  attendant 
contracts  identified  by  year  over  a  several-year  period.  We  use  this  dataset,  when 
combined,  for  cross-sectional,  time-series  analysis,  also  known  as  panel  data  analysis 
(Princeton  University,  2007). 

Following  the  descriptive  statistical  analysis,  we  employ  a  linear  multiple 
regression  analysis  that  combines  cross-sectional,  time-series  SAR  data  by  examining  the 
effect  of  specified  explanatory  variables  on  an  outcome  measure.  Additionally,  we 
propose  future  research  using  structural  equation  modeling  (SEM),  a  method  that  allows 
the  simultaneous  analysis  of  multiple  factors.  SEM  analysis  permits  the  inclusion  of 
conceptually  constructed  variables  that  underlie  the  measurable  variables,  thereby, 
helping  to  explain  the  full  effect  of  fixed-price  incentive  R&D  contracts  within  MDAPs 
on  later  period  cost  and  schedule  variance  during  the  production  and  deployment  phase. 
The  use  of  a  structural  equation  model  would  allows  others  to  examine  variance  in  ways 
not  previously  available;  that  is,  it  would  allow  the  consideration  of  multiple,  interrelated, 
and  simultaneously  varying  factors,  such  as  the  cost,  schedule,  and  engineering  variances 
of  a  program  and  the  earned  value  cost  and  schedule  variances  of  that  program’s 
contracts. 


As  indicated,  our  objective  in  this  research  is  to  determine  the  likely  effects  of  the 
increased  usage  of  fixed-price  R&D  contracts  during  EMD  on  long-term  program  cost, 
schedule,  and  engineering  variances.  Although  we  give  SARs  particular  emphasis,  we 
also  utilize  DAES  reports  and  Federal  Procurement  Data  System  (FPDS)  data  from  the 
Defense  Acquisition  Management  Information  Retrieval  System  (DAMIRS).  The 
methodology  we  used  in  this  research  consisted  of  the  following  steps: 


1 .  conduct  a  literature  review; 

2.  collect  historical  data  from  DAMIRS; 

3.  build  a  database  that  includes  both  program  data  and  contract  data  by  year, 
with  all  required  fields  in  sufficient  number  for  analysis  using  SPSS  (a 
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statistical  package  that  permits  cross-sectional,  time-series  analysis  of 
measurement  variables); 

4.  conduct  a  descriptive  statistical  analysis; 

5.  complete  an  initial  model  for  CV,  SV,  and  EV; 

6.  complete  an  initial  model  pooling  of  CV,  SV,  EV,  and  other  relevant 
program,  cost,  and  schedule  contract  variances  measured  in  dollars; 

7.  conduct  a  thorough  investigation  of  the  program  and  contract  variances 
using  regression  analysis  to  identify  explanatory  variables  that  influence 
these  measures;  and 

8.  prepare  a  recommendation  regarding  R&D  contract  type  selection. 


E.  ORGANIZATION  OF  STUDY 


In  Chapter  II  (Background),  we  provide  an  explanation  of  the  current  DoD 
acquisition  process,  the  DoD  acquisition  categories,  and  different  contract  types.  We 
provide  a  summary  of  the  relationship  between  contract  types  and  the  acquisition  phases, 
and  a  description  of  the  possible  risk  experienced  during  different  phases  of  a  program. 
In  this  chapter,  we  also  include  sections  discussing  the  importance  of  R&D  along  with 
technology  readiness  levels. 

In  Chapter  III  (Purpose),  we  provide  explanations  of  the  regulations  that  govern 
acquisition  variances  and  aid  in  the  decision-making  of  acquisition  professionals, 
including  the  Nunn-McCurdy  Amendment,  Acquisition  Program  Baselines  (APBs), 
SARs,  and  Earned  Value  Management.  We  include  a  definition  of  each  of  the  program 
and  contract  variances  utilized  in  this  research.  We  also  provide  in  Chapter  III  the 
information  required  to  understand  program-  and  contract-level  variances,  including 
program  cost,  schedule,  and  engineering  variances,  as  well  as  contract  earned  value  cost 
and  schedule  variances.  We  conclude  the  chapter  with  an  explanation  of  why  the  study 
of  these  acquisition  variances  is  important. 


In  Chapter  IV  (Database  Construction  and  Analysis),  we  describe  the  process  we 


used  to  collect  data  and  to  perform  the  analysis  of  variance.  The  chapter  contains  the 
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steps  we  followed,  including  our  data  collection  methods  and  the  use  of  descriptive 
statistics. 

In  Chapter  V  (Statistical  Methodology  and  Results),  we  provide  an  explanation  of 
our  methodology  and  analysis.  The  regression  analysis  methodology  is  discussed.  This 
analysis,  conducted  using  cross-section,  time-series  methods,  should  be  viewed  as 
exploratory,  and  certain  aspects  of  our  results  need  to  be  interpreted  in  this  light.  This 
includes  our  methodology  for  cross-sectional,  time-series  analysis.  Our  analysis  includes 
identifying  the  relationship  between  preproduction  program  and  contract  variances,  and 
future  program  variances.  The  interrelationships  we  describe  will  help  acquisition 
professionals  better  understand  both  the  effects  of  cost,  schedule,  and  engineering 
variances  on  programs,  and  the  linkage  between  programs. 

In  Chapter  VI  (Summary,  Conclusions,  and  Recommendations),  we  explain  the 
findings  of  our  research  and  discuss  the  implications  of  the  findings  to  practitioners.  We 
then  draw  conclusions  based  on  those  results.  We  also  provide  recommendations  for 
future  research  efforts  and  disclose  the  limitations  of  the  study. 

F.  BENEFITS  OF  THE  STUDY 

We  anticipate  that  this  research  will  help  acquisition  personnel  develop  a  better 
understanding  of  program  and  contract  cost  and  schedule  variances,  and  possibly  provide 
indications  of  future  program  cost  increases  to  establish  better  management  techniques. 
Understanding  the  implications  of  fixed-price  incentive  R&D  acquisition  programs  on 
cost,  schedule,  and  engineering  variances  will  help  the  DoD  set  appropriate  policies  for 
contract  type  in  the  early  stages  of  system  development.  In  this  research  report,  we 
provide  the  first  comprehensive  analysis  of  the  relationship  between  program  data  and 
contract  data.  One  reason  why  understanding  this  relationship  is  so  important  is  that 
problems  with  cost  growth  and  schedule  slippage  are  typically  reported  at  the  program 
level  but  the  contract  type  applies  at  the  contract  level,  and  a  program  may  contain 
several  different  contracts  during  both  the  EMD  phase  and  the  production  phase.  This 
research  will  also  help  the  DoD  more  appropriately  consider  the  long-term  impacts  of 
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proposed  procurement  cuts  and  will  help  leaders  determine  the  best  structure  of  those 
cuts.  It  may  also  improve  system  delivery  time  to  the  warfighter. 
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II.  BACKGROUND 


A.  OVERVIEW 

In  this  chapter,  we  provide  the  background  information  required  to  understand  the 
scope  of  this  research,  including  a  review  of  the  DoD  acquisition  process  and  three 
decision  support  systems.  We  briefly  explain  federal  contract  types  applicable  to 
acquisitions,  along  with  the  preferred  contract  types  for  each  phase  of  the  DoD 
acquisition  process.  Then,  we  explore  the  importance  of  R&D  to  acquisition  program 
success,  and  we  discuss  technology  readiness  levels.  The  risk  associated  with  contract 
type  and  acquisition  phase  is  reviewed.  We  conclude  the  chapter  with  a  discussion  of  the 
primary  documents  from  which  the  study  draws  data. 

B.  DEFENSE  ACQUISITION  PROCESS 

In  this  section,  we  provide  an  overview  of  the  DoD  acquisition  process  and  the 
three  principal  decision  support  systems:  the  Planning,  Programming,  Budgeting,  and 
Execution  (PPBE)  process;  the  Joint  Capabilities  Integration  and  Development  System 
(JCIDS);  and  the  Defense  Acquisition  System.  These  three  integrated  systems  ensure 
acquisition  personnel  make  consistent  decisions  to  efficiently  acquire  the  right  products 
and  services  required  by  the  warfighter.  The  DoD’s  decision  support  systems,  described 
in  the  Defense  Acquisition  Guidebook  (DoD,  2011),  and  its  policies,  oversight,  and 
integration  can  be  seen  in  Figure  1.  The  boxes  in  Figure  1  annotate  the  direct  supporting 
references  for  each  system.  The  overlap  in  the  three  systems  can  clearly  be  seen  in  the 
figure  and  is  important  to  understanding  the  balance  of  all  three  systems  in  the  defense 
acquisition  process. 
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Figure  1.  The  DoD  Decision  Support  Systems 

(DoD,  2011) 

1.  The  Planning,  Programming,  Budget,  and  Execution  (PPBE)  Process 

The  PPBE  process  provides  for  strategic  planning,  program  development,  and 
resource  determination.  The  PPBE  process  generates  funding  appropriations  to  support 
DoD  acquisition  requirements.  During  this  process,  the  Secretary  of  Defense  provides 
priorities,  which  guide  resource  allocation  decisions.  Program  managers  must  be  aware 
of  this  process  to  effectively  manage  the  funding  of  contracts.  Failure  to  recognize  the 
link  between  budgets  and  requirements  in  the  acquisition  process  may  directly  result  in 
poor  performance  during  the  acquisition  process  (Gansler,  2002). 

Accurate  program  estimates  are  important  in  order  to  obtain  necessary  funding 
during  the  budgeting  process.  The  timing  of  funding  availability  is  also  important  and 
could  significantly  affect  program  schedules  and  costs.  In  an  address  to  the  Senate 
Armed  Services  Committee  regarding  the  FY2011  Continuing  Resolution,  Secretary  of 
Defense  Robert  Gates  stated  that  the  failure  of  timely  funding  would  damage 
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procurement  and  research  programs,  causing  delays,  increasing  costs,  and  disrupting 
production  of  highly  demanded  assets  (Gates,  2011). 

2.  Joint  Capabilities  Integration  and  Development  System  (JCIDS) 
Process 

The  PPBE  process  receives  development  and  production  life  cycle  estimates  from 
the  requirements  defined  in  the  Joint  Capabilities  Integration  and  Development  System 
(JCIDS)  process.  JCIDS  procedures  support  the  Chairman  of  the  Joint  Chiefs  of  Staff 
(CJCS)  and  the  Joint  Requirements  Oversight  Council  (JROC)  in  determining  joint 
military  capability  needs.  JCIDS  is  the  DoD’s  systematic  means  for  appraising  gaps  in 
military  warfighting  capabilities  and  for  proposing  solutions  to  solve  those  gaps. 
Announced  in  2003,  the  JCIDS  process  changed  the  way  requirements  were  identified — 
replacing  Service-centric  requirements  processes  with  a  joint  capabilities  system.  The 
change  came  after  Secretary  of  Defense  Donald  Rumsfeld  sent  a  memo  that  stated, 

Please  think  through  what  we  all  need  to  do  ...  to  get  the  requirements 
system  fixed.  It  is  pretty  clear  it  is  broken,  and  it  is  so  powerful  and 
inexorable  that  it  invariably  continues  to  require  things  that  ought  not  to  be 
required,  and  does  not  require  things  that  need  to  be  required.  (CJCS, 

2006,  p.  5) 

The  elimination  of  the  word  requirements  from  the  generation  system  signaled  the 
DoD’s  intent  for  the  JCIDS  process  to  determine  possible  procedural  or  training-based 
solutions  (non-materiel  solutions)  available  along  with  materiel  solutions  and  to  justify 
the  need  for  the  capability  (CJCS,  2006). 

The  JCIDS  process  is  covered  in  CJCS  Instruction  3170.01G  (CJCS,  2009)  and  is 
summarized  here  to  provide  for  an  understanding  of  the  requirements  review  process  in 
defense  acquisitions.  The  process  implemented  in  JCIDS  supports  the  management  of 
resource  investments  and  must  be  performed  prior  to  the  acquisition  process  for  new 
systems  to  begin.  Figure  2  illustrates  the  acquisition  process  and  what  documents  are 
produced  during  the  JCIDS  process  in  order  to  identify  gaps  in  capabilities  (CJCS,  2009). 
As  seen  in  Figure  2,  various  parties  are  involved  in  the  process  and  are  required  to 
produce  multiple  coordinating  documents. 
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□  =  Sponsor  Activity  JCIDS  Document  OA-  Acquisition  decision 

Figure  2.  JCIDS  and  Defense  Acquisition 

(CJCS,  2009) 

The  JCIDS  process  incorporates  the  military  missions  and  the  capabilities 
required  to  perform  the  operational  objectives  associated  with  these  missions.  The 
incorporation  of  capabilities  into  the  JCIDS  process  originates  through  a  capabilities- 
based  assessment  (CBA)  and  an  identified  valid  gap  in  mission  requirement.  As  stated  by 
General  McChrystal, 

If  only  non-materiel  solutions  are  recommended  or  a  non-materiel  solution 
can  be  implemented  independent  of  proposed  materiel  needs,  a  joint 
doctrine,  organization,  training,  materiel,  leadership  and  education, 
personnel,  or  facilities  (DOTMLPF)  change  recommendation  (DCR)  is 
produced.  (CJCS,  2009) 

An  initial  capabilities  document  (ICD)  identifies  gaps  requiring  materiel  solutions.  An 
approved  ICD  provides  the  required  information  to  form  the  materiel  development 
decision  (MDD)  and  starts  the  actual  acquisition  process  (CJCS,  2009;  see  Figure  2). 

3.  Defense  Acquisition  System  Process 

The  objective  of  the  defense  acquisition  system  is  to  acquire  products  that  satisfy 
specified  needs  and  provide  measurable  improvements  to  mission  capabilities  at  a  fair 
and  reasonable  price  (DoD,  201 1).  Defense  acquisition  follows  distinct  program  phases, 
a  series  of  decision  points,  and  significant  milestones.  The  process  defined  in  the  DoD 
5000  Series  that  guides  acquisition  programs  for  MDAPs  is  summarized  here.  Figure  3 
illustrates  the  defense  acquisition  system  process  by  showing  the  relationship  between 
each  phase  and  milestone. 
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Figure  3.  The  Defense  Acquisition  Management  System 

(USD[AT&L],  2008) 

a.  Materiel  Solution  Analysis  Phase  and  Milestone  A 

After  an  approved  ICD  and  an  MDD,  the  milestone  decision  authority 
(MDA)  may  authorize  entry  into  the  acquisition  management  system,  starting  with  the 
materiel  solution  analysis  phase,  also  known  as  the  concept  refinement  phase  (see  Figure 
3).  For  MDAPs,  a  defense  acquisition  board  (DAB)  is  formed  that  provides  advice  on 
critical  acquisition  decisions.  The  DAB  is  chaired  by  the  USD(AT&L)  and  includes 
senior  officials  from  the  Joint  Staff,  the  military  departments,  and  the  staff  offices  within 
the  Office  of  the  Secretary  of  Defense  (OSD).  The  USD(AT&L)  is  the  MDA  when  there 
is  a  DAB.  The  purpose  of  the  materiel  solution  analysis  phase  is  to  perform  an  analysis 
of  alternatives  (AoA)  for  all  potential  materiel  solutions;  the  phase  culminates  with  a 
decision  about  whether  to  proceed  with  MS  A  as  designated  by  the  MDA.  An  AoA 
assesses  each  alternative  on  measures  of  effectiveness,  cost,  schedule,  concept  of 
operations,  and  overall  risk.  The  materiel  solution  analysis  phase  ends  once  the  MDA 
approves  the  results  of  the  completed  AoA  (USD[AT&L],  2008). 
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b.  Technology  Development  Phase  and  Milestone  B 

The  acquisition  decision  memorandum  is  signed  at  MS  A,  documenting  an 
approved  materiel  solution,  a  technology  development  strategy  (TDS),  and  entry  into  the 
technology  development  phase.  The  TDS  establishes  the  preliminary  acquisition  plan, 
including  the  cost,  schedule,  and  performance  goals  for  the  engineering  and 
manufacturing  development  phase.  The  purpose  of  the  technology  development  phase  is 
to  reduce  technology  risk  and  to  establish  subsystems  and  components  that  must  be 
demonstrated  before  being  fully  integrated  into  a  system  (USD[AT&L],  2008). 

This  phase  is  a  continuous  discovery  and  development  process  to  identify 
the  accessibility  and  refinement  of  the  requirement.  Competitive  prototypes  are  built 
based  on  initial  capabilities.  The  users,  or  the  relevant  Service,  should  establish  a 
capabilities  development  document  (CDD)  during  the  technology  development  phase  that 
specifies  the  technical  performance  parameters  required  to  deliver  the  proposed  design 
and  that  fills  the  capabilities  gap  identified  in  the  ICD. 

The  CDD  replaced  the  operational  requirements  document  (ORD)  that 
was  used  under  the  old  requirements  system.  The  CDD  supports  the  MS  B  decision  and 
must  be  approved  prior  to  MS  B.  It  guides  the  engineering  and  manufacturing 
development  phase  by  defining  measurable  and  testable  capabilities.  The  CDD  identifies 
operational  performance  attributes  of  the  proposed  system  known  as  key  performance 
parameters  (KPPs).  KPPs  make  up  the  bulk  of  the  CDD  and  list  each  required  measure 
of  effectiveness.  The  parameters  contain  both  threshold  (required  or  minimum)  and 
objective  (desired  or  maximum)  performance  values  (USD[AT&L],  2008). 

The  program  manager  should  prepare  for  a  preliminary  design  review 
(PDR)  prior  to  MS  B,  as  planned  by  the  development  strategy.  The  system  developers, 
engineers,  users,  and  certification  authorities  should  all  collaborate  to  agree  on  a 
proposed  solution  based  on  demonstrated  technology  that  meets  the  PDR.  The  PDR 
report  and  the  demonstrated  manufacturing  process  are  provided  to  the  MDA  at  MS  B  to 
identify  projected  cost,  schedule,  and  risk.  At  MS  B,  the  MDA  decides  to  proceed  to  the 
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next  acquisition  phase,  perform  additional  work,  or  terminate  the  effort  (USD[AT&L], 
2008). 


c.  Engineering  and  Manufacturing  Development  Phase  and 
Milestone  C 

The  engineering  and  manufacturing  development  (EMD)  phase  begins 
with  an  approved  acquisition  strategy  at  MS  B.  MS  B  and  entrance  into  the  EMD  phase 
typically  mark  the  initiation  of  an  acquisition  program.  The  purpose  of  the  EMD  phase  is 
to  develop  an  integrated  system  of  demonstrated  subsystems  and  components  design.  In 
this  phase,  a  proven  system  capability  is  also  developed,  and  an  achievable  and 
affordable  manufacturing  process  is  demonstrated  (USD[AT&L],  2008). 

The  integrated  system  design  defines  functionality  and  interfaces;  provides 
a  complete,  detailed  design;  and  reduces  full  system-level  risk.  The  system  capability 
and  manufacturing  process  demonstration  assures  that  the  system  will  operate  in 
accordance  with  performance  parameters  and  demonstrates  that  system  production  can  be 
supported.  This  phase  ends  when  the  system  meets  all  approved  requirements,  performs 
in  its  intended  environment,  effectively  demonstrates  manufacturing  capabilities,  has 
reasonably  available  production  capabilities,  meets  MS  C  requirements,  and  has  the 
approval  of  the  MDA  to  commit  the  program  or  to  terminate  the  effort  (USD[AT&L], 
2008). 


d.  Production  and  Deployment  Phase 


The  production  and  deployment  phase  is  the  final  phase  in  the  systems 
acquisition  process  (see  Figure  3).  The  purpose  of  this  phase,  according  to  DoD 
Instruction  5000.02  (USD[AT&L],  2008),  is  “to  achieve  an  operational  capability  that 
satisfies  mission  needs”  (p.  26).  MS  C  approval  begins  the  production  and  deployment 
phase.  MS  C  specifically  authorizes  low-rate  initial  production  (LRIP),  which  is 
performed  at  the  beginning  of  the  phase.  In  the  past,  the  production  and  deployment 
phase  did  not  begin  until  the  start  of  full-scale  production,  at  MS  III.  For  the  purposes  of 
this  research,  we  considered  the  beginning  of  the  production  and  deployment  phase  as 
occurring  at  LRIP  approval,  even  for  programs  that  did  not  have  a  milestone  review  at 
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that  point.  We  provide  further  information  regarding  our  data  collection  methods  and 
analysis  in  later  chapters.  During  LRIP,  manufacturing  capability  is  verified.  Initial 
operational  testing  and  evaluation  is  performed  prior  to  the  point  that  the  MDA 
authorizes  entry  into  full-rate  production  and  deployment  from  a  successful  full-rate 
production  decision  review  (USD[AT&L],  2008). 

4.  Acquisition  Program  Categories 

Acquisition  programs  are  assigned  to  an  acquisition  category  (ACAT)  based  on 
their  location  in  the  acquisition  process,  on  cost,  or  on  whether  the  program  is  of  special 
interest  to  the  MDA.  The  USD(AT&L)  can  reclassify  an  acquisition  program  at  any 
time.  The  discussion  in  the  following  subsections  explains  the  differences  between  each 
category. 


a.  ACA  T I  Programs 

ACAT  I  programs,  also  called  Major  Defense  Acquisition  Programs 
(MDAPs),  include  programs  designated  by  the  MDA  as  special  interest.  According  to  10 
U.S.C.  §  2430  (201 1),  MDAPs  are  designated  by  the  Secretary  of  Defense  and  estimated 
to  require  a  total  expenditure  for  RDT&E  of  more  than  $300  million  in  FY1990  dollars  or 
for  procurement  of  more  than  $1.8  billion  in  FY1990  dollars.  ACAT  I  programs  are 
further  divided  by  decision  authority.  The  programs  with  the  highest  interest  are  ACAT 
ID,  and  the  MDA  is  the  USD(AT&L).  For  ACAT  IC,  the  MDA  is  head  of  the  DoD 
Component  or,  if  delegated,  the  Component  Acquisition  Executive  (CAE;  USD[AT&L], 
2008). 


b.  A  CA  T I  A  Programs 


ACAT  I A  programs  are  also  called  Major  Automated  Information 
Systems  (MAIS)  and,  in  some  cases,  meet  the  definition  of  an  MDAP,  but  also  include 
programs  designated  by  the  MDA  as  special  interest  programs.  According  to  10  U.S.C.  § 
2445  (2011),  MAIS  programs  are  designated  by  the  Secretary  of  Defense  when  program 
expenditures  in  FY2000  constant  dollars  are  estimated  to  exceed  the  following  amounts: 
$32  million  for  all  expenditures  directly  related  to  automated  information  systems 
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definition,  design,  development,  and  deployment  costs  in  a  single  FY;  $126  million  for 
all  expenditures  directly  related  to  the  entire  program;  or  $378  million  for  all 
expenditures  related  to  the  total  life  cycle  costs.  ACAT  IA  programs  are  further  divided 
into  ACAT  IAM,  in  which  the  MDA  is  the  USD(AT&L)  or  his  or  her  designee,  and 
ACAT  I  AC,  in  which  the  MDA  is  the  head  of  the  DoD  Component  or,  if  delegated,  the 
CAE  (USD[AT&L],  2008).  In  this  research,  we  examine  only  MAISs  that  are  also 
MDAPs. 


c.  ACAT  II  and  ACAT  III  Programs 

ACAT  II  programs  do  not  meet  criteria  for  ACAT  I,  but  they  are  still 
major  systems  with  a  dollar  value  in  FY2000  constant  dollars  estimated  by  the  DoD 
Component  Head  of  eventual  RDT&E  greater  than  $140  million  or  of  procurement  of 
more  than  $660  million.  The  MDA  for  ACAT  II  programs  is  the  CAE  or  an  individual 
designated  by  the  CAE.  ACAT  III  programs  include  DoD  programs  that  do  not  meet 
ACAT  II  criteria,  and  the  MDA  for  these  programs  is  designated  by  the  CAE 
(USD[AT&L],  2008).  We  do  not  directly  examine  ACAT  II  and  ACAT  III  programs  in 
this  research,  although  it  may  be  possible  to  generalize  some  of  the  findings  of  this 
research  to  these  programs  when  sufficient  similarity  in  treatment  exists. 

C.  FEDERAL  CONTRACT  TYPES 

A  wide  variety  of  contracts  can  be  used  to  purchase  products  and  services 
required  by  the  federal  government.  A  large  acquisition  program  typically  has  multiple 
types  of  contracts,  and  selecting  the  right  contract  type  for  an  acquisition  is  essential  to 
successful  program  completion.  Federal  Acquisition  Regulation  (FAR,  2011)  part  16 
describes  the  different  federal  contract  types.  Contracts  vary  according  to  the 
responsibility  assumed  by  the  contractor  for  the  costs  of  performance  and  the  amount  and 
nature  of  profit  incentives  offered  to  the  contractor  for  specified  standards  or  goals  (FAR, 
2011).  For  DoD  MDAPs  and  this  research,  the  two  general  categories  of  contracts  are 
fixed  price  and  cost  reimbursement,  but  there  are  other  types  outside  these  two 
categories. 
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FIXED-PRICE  CONTRACTS 


Fixed-price  contracts  are  suitable  when  acquiring  supplies  and  services  that  users 
can  describe  in  sufficient  detail.  The  price  is  agreed  upon  during  the  award  phase,  or,  in 
appropriate  circumstances,  an  adjustable  price  may  be  included.  Fixed-price  contracts 
providing  an  adjustable  price  may  include  a  ceiling  price,  a  target  price,  or  a  combination 
of  both.  Under  a  fixed-price  contract,  most  of  the  performance  and  cost  risks  are  placed 
on  the  contractor  through  the  use  of  incentives  to  control  costs  (FAR,  2011).  The 
following  summaries  of  the  different  types  of  fixed-price  contracts  are  based  on  FAR 
(2011)  part  16. 


a.  Firm-Fixed-Price  Contracts 

Firm- fixed-price  (FFP)  contracts  are  used  when  a  fair  and  reasonable  price 
can  be  determined  at  the  beginning  of  the  contract  (FAR,  2011).  The  government  pays 
the  negotiated  amount  regardless  of  the  contractor’s  actual  performance  costs. 
Administrative  requirements  are  not  eliminated  for  FFP,  but  the  burden  is  reduced. 
Given  well-defined  specifications,  the  contractor  bears  most  of  the  risk.  This  contract 
type  is  preferred  above  all  others  because  it  encourages  the  contractor  to  contain  costs 
(Garrett,  2007).  The  government  prefers  FFP  if  clear  objectives  exist  and  accurate 
pricing  data  is  available,  but  that  may  not  always  be  the  case. 


b.  Fixed-Price  Incentive  Contracts 


Fixed-price  incentive  (FPI)  contracts  are  used  when  the  government  wants 
to  incentivize  technical  performance  and  cost  controls.  Parties  can  negotiate  a  target  cost, 
target  profit,  and  a  ceiling  price  that  provides  for  the  contractor  to  assume  an  appropriate 
share  of  the  risk.  If  the  contractor  reaches  the  ceiling  price  as  a  result  of  an  overrun,  the 
contract  essentially  becomes  an  FFP  contract.  In  principle,  the  contractor  is  paid  no  more 
than  the  ceiling  price  and  must  meet  the  requirements  of  the  contract.  The  profit  is 
adjusted  by  calculating  the  final  price  using  a  formula  based  on  the  relationship  of  final 
negotiated  and  target  costs.  Two  different  types  of  fixed-price  incentive  contracts  are 
authorized:  fixed-price  incentive  firm  (FPIF)  target  contracts  and  fixed-price  incentive 
successive  (FPIS)  targets  contracts  (FAR,  2011).  Currently,  there  is  a  proposed  Defense 
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Federal  Acquisition  Regulation  Supplement  (DFARS,  2011)  rule  to  encourage  the 
increased  use  of  fixed-price  incentive  contracts.  The  origination  of  this  rule  is  credited  to 
a  memo  sent  by  the  USD(AT&L)  on  November  3,  2010  (Levin,  J.,  2011),  that  gave 
direction  on  attaining  better  efficiency  and  productivity  in  defense  contract  spending.  In 
this  study,  we  analyze  this  type  of  contract  closely  and  identify  the  size  of  variances 
occurring  with  this  type  of  contract  compared  to  other  types  of  contracts. 

2.  COST-REIMBURSEMENT  CONTRACTS 

Cost-reimbursement  contracts  allow  the  government  to  pay  the  contractor  all 
allowable  incurred  costs  that  are  fair  and  reasonable  as  prescribed  in  the  contract.  This 
type  of  contract  is  used  when  many  uncertainties  associated  with  contract  performance 
and  costs  cannot  be  estimated  with  sufficient  accuracy  to  use  a  fixed-price  contract.  A 
cost-reimbursement  contract  provides  an  estimate  of  the  total  costs  for  the  purpose  of 
obligating  funds,  and  it  establishes  funding  ceilings  that  cannot  be  exceeded  without 
approval  from  the  contracting  officer.  Contractors  may  exceed  these  ceilings  at  their  own 
risk.  Cost-reimbursement  contracts  are  typically  renegotiated  or  terminated  if  total  costs 
exceed  ceilings. 

Cost-type  contracts  place  most  of  the  cost  and  performance  risk  on  the 
government  (FAR,  2011).  The  contractor  should  put  forth  its  best  effort,  but  there  is  no 
promise  of  results.  The  government  may  not  end  up  with  the  final  product  or  service  it 
contracted  for,  but  it  must  reimburse  the  contractor  for  costs  incurred  that  do  not  exceed 
the  approved  funding  ceilings.  The  government  is  obligated  to  reimburse  all  actual  costs 
that  are  allowable,  allocable,  and  reasonably  incurred  to  the  extent  prescribed  in  the 
contract  (FAR,  201 1).  In  the  following  subsections  we  summarize  the  different  types  of 
cost  contracts;  these  summaries  are  based  on  FAR  (2011)  part  16. 

a.  Cost-Plus-Fixed-Fee  Contracts 

Cost-plus-fixed-fee  (CPFF)  contracts  pay  a  pre-determined,  fixed-fee  that 
is  agreed  upon  during  contract  negotiations.  The  fee  may  be  adjusted  for  changes  in  the 
work  to  be  performed  (FAR,  2011).  The  contracting  officer  is  responsible  for  monitoring 
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the  contractor’s  expenditures  and  can  request  an  audit  of  the  contractor’s  vouchers  at  any 
time,  similar  to  other  cost-reimbursement  auditing. 

b.  Cost-Plus-Incentive-Fee  Contracts 

Cost-plus-incentive-fee  (CPIF)  contracts  are  used  to  encourage  contractors 
by  providing  greater  profits  through  cost  savings  and  other  performance  improvements. 
The  government  pays  allowable,  reasonable,  and  allocable  costs  and  an  incentive  fee 
based  on  the  contractor’s  achievement  of  the  objectives  calculated  by  using  a  formula 
that  is  based  on  the  relationship  of  total  allowable  and  target  costs.  The  contract  specifies 
target  costs  and  target  fees  and  establishes  minimum  and  maximum  fees  and  a  fee- 
adjustment  formula.  The  increase  or  decrease  in  fee  is  intended  to  incentivize  the 
contractor  to  effectively  manage  costs  (FAR,  2011). 

c.  Cost-Plus-Award-Fee  Contracts 

Cost-plus-award-fee  (CPAF)  contracts  are  used  to  provide  additional 
incentives  to  contractors  to  achieve  excellence  in  areas  such  as  quality,  timeliness, 
technical  ingenuity,  and  cost-effective  management  (FAR,  2011).  The  government  pays 
allowable  costs,  a  base  fee,  and  an  award  fee  based  on  a  subjective  evaluation  of 
performance.  The  contract  provides  for  interim  rating  periods  during  contract 
performance.  It  is  possible  that  the  inclusion  of  KPPs  in  an  MDAP’s  CDD  at  MS  B  may 
induce  the  program  manager  to  incentivize  KPPs  through  the  use  of  award  fee  incentives 
(Hildebrandt,  2010)  since  award  fees  offer  the  ability  to  incentivize  ambiguously  defined 
objectives. 


D.  TYPICAL  CONTRACT  TYPE  BY  ACQUISITION  PHASE 


The  government  can  use  a  wide  variety  of  contracts  to  order  required  products 
and  services.  Selection  of  the  contract  type  is  driven  by  risk  considerations  (FAR,  2011), 
which  vary  among  programs  and  across  acquisition  life  cycle  phases  (Garrett,  2007). 
Certain  contract  types  are  more  suited  to  certain  types  of  acquisitions  and  certain  stages 
of  the  acquisition  life  cycle  (FAR,  2011;  Defense  Acquisition  University  [DAU],  2008). 
No  particular  contract  type  is  the  answer  for  any  phase,  but  enduring  trends  in  selection 
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of  contract  type  exist  due  to  the  limitations  typically  present  during  each  life  cycle  stage. 
Preferred  contract  types  for  a  phase  may  shift  over  time  due  to  changing  policy  or 
guidance  or  to  shifting  practitioner  preference  based  on  research  regarding  programs  of  a 
similar  nature.  As  we  explain  in  more  detail  later  in  this  section,  contract  type  is 
ultimately  a  collaborative  decision  between  a  program’s  contracting  officer  and  program 
manager,  subject  to  negotiation  with  a  program’s  contractors  (FAR,  2011;  DFARS, 
2011). 

1.  RISK  AND  PREFERRED  CONTRACT  TYPE 

Fixed-price  contracts  are  typically  preferred  by  the  government  to  minimize  its 
risk,  but  they  may  not  be  appropriate  if  the  work  lacks  precise  specifications  or  cost 
estimates.  FAR  (2011)  subpart  16.101  characterizes  contract  types  as  those  ranging 
“from  firm- fixed-price,  in  which  the  contractor  has  full  responsibility  for  the  performance 
costs  and  resulting  profit  (or  loss),  to  cost-plus-fixed-fee,  in  which  the  contractor  has 
minimal  responsibility  for  the  performance  costs  and  the  negotiated  fee  (profit)  is  fixed.” 
If  funding  ends  before  completion,  the  contract  should  be  renegotiated  or  terminated. 
Incentive  contracts  fall  in  between  these  two  extremes.  Fixed-price  incentive  and  cost- 
plus-incentive-fee  contracts  should  only  be  considered  if  it  is  in  the  government’s  best 
interest  to  use  cost  and,  when  appropriate,  performance  incentives  (FAR,  2011). 

Cost  estimates  are  ultimately  predictions  about  the  future,  and  they  are  subject  to 
error  (Garrett,  2007).  Buyers  and  sellers  understand  that  error  is  present  in  cost 
estimation.  For  the  purposes  of  this  research,  we  consider  the  government  the  buyer  and 
the  contractors  the  sellers.  The  government  is  wary  of  accepting  an  excessive  cost  for 
goods  or  services,  and  contractors  seek  to  ensure  that  they  do  not  suffer  a  loss  on  a  sale. 
Each  party  attempts  to  hedge  against  its  own  risk.  Contractors  estimate  pessimistically, 
and  the  government  estimates  optimistically,  resulting  in  a  “range  of  possible  costs” 
(Garrett,  2007,  p.  106). 

Whenever  practicable,  the  government’s  default  option  is  to  use  a  firm- fixed-price 
contract,  but  this  is  frequently  not  possible.  If  a  large  amount  of  estimating  uncertainty  is 
present,  the  government  and  a  contractor  may  be  unable  to  reach  an  agreement  without 
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the  government  accepting  an  excessively  pessimistic  cost  estimate.  This  results  in  the 
government  negotiating  with  a  contractor  to  establish  the  contract’s  type  and  price.  This 
negotiation  often  results  in  the  government  sharing  cost  risk  with  the  contractor  to  bring 
down  the  contracted  price,  although  achieving  a  more  reasonable  price  is  not  the  sole 
objective.  The  objective  of  this  negotiation  “is  to  negotiate  a  contract  type  and  price  (or 
estimated  cost  and  fee)  that  will  result  in  reasonable  contractor  risk  and  provide  the 
contractor  with  the  greatest  incentive  for  efficient  and  economical  performance”  (FAR, 
2011,  subpart  16.103).  Selection  of  contract  type  can  be  depicted  as  a  trade-off  between 
the  buyer’s  risk  (risk  to  the  government)  and  the  seller’s  risk  (risk  to  the  contractor),  as 
shown  in  Figure  4. 


Types  of  Contracts 


Fixed-price 


Cost-reimbursement 


4 


FFP 


- ► 

FP/EPA  FPI  CS  CR  CPIF  CPAF  CPFF 


Low  ◄ -  Buyer’s  risk  - ►  High 

High  ◄ -  Seller’s  risk  - ►  Low 

Figure  4.  Types  of  Contracts  by  Risk 

This  figure  from  Garrett  (2007,  p.  127)  was  altered  to  fit  the  context  of  this 
research,  federal  MDAPs.  Accordingly,  the  Time  and  Materials  contract  type  was 
eliminated  from  the  graphic;  although  it  is  a  permissible  federal  contract  type,  it  is  not 
significant  with  respect  to  MDAPs.  The  Cost  Plus  Percentage  of  Cost  contract  type  was 
also  eliminated  because,  by  law,  this  contract  type  is  no  longer  permissible  for  federal 
contracts. 

2.  RISK  BY  ACQUISITION  PHASE 

During  the  beginning  stages  of  system  acquisition,  risk  is  typically  higher. 
System  acquisition  risk  is  initially  driven  by  a  lack  of  program  definition  and  by 
uncertainty  regarding  the  program’s  ability  to  meet  scheduled  technical  achievements. 
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CDD  is  an  attempt  to  change  this.  From  a  contracting  perspective,  risk  can  be  understood 
as  poorly  or  undefined  requirements  driving  uncertainty  in  performance  cost.  When 
looking  beyond  a  contracting  risk  perspective,  acquisition  risk  can  be  understood  as  a 
combination  of  four  factors:  technical  risk,  cost  risk,  schedule  risk,  and  programmatic 
risk  (Blanchard  &  Fabrycky,  2006). 

Technical  risk  is  the  possibility  that  a  technical  requirement  may  not  be  met 
during  a  system’s  life  cycle  (International  Council  on  Systems  Engineering  [INCOSE], 
2004).  For  a  government  acquisition,  this  could  be  understood  as  a  program’s  failure  to 
achieve  one  or  more  of  the  threshold  requirements.  Cost  risk  is  the  possibility  that  the 
specific  budget  allocated  to  a  program  will  be  exceeded  (Blanchard  &  Fabrycky,  2006; 
INCOSE,  2004).  Schedule  risk  is  the  possibility  that  a  program  will  fail  to  meet 
scheduled  milestones,  and  programmatic  risk  is  the  external  risk  posed  by  the  program’s 
environment  (INCOSE,  2004).  Common  causes  for  program  cancellation  are  cost 
overruns,  schedule  slips,  and  failure  to  eliminate  technological  risk.  Figure  5  depicts 
relationships  between  these  risks,  as  proposed  by  the  INCOSE. 


Figure  5.  Typical  Relationships  Among  the  Risk  Categories 

(INCOSE,  2004,  p.  65) 
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Technical  risk  is  typically  highest  at  the  beginning  of  a  program,  when  a 
program’s  technical  problems  are  least  understood  (INCOSE,  2004).  Cost  risk  also 
typically  decreases  as  requirements  become  more  defined  and  as  a  program  gains 
institutional  support.  This  increase  in  support  protects  maturing  programs  from  budget 
reductions.  Schedule  risk  also  normally  decreases  as  a  program  develops,  although  the 
effect  of  a  schedule  slip  increases  as  more  resources  are  devoted  to  a  program.  FAR 
(201 1)  subpart  16.104(d)  recognizes  that  acquisition  risk  is  highest  at  the  beginning  of  a 
program: 


Complex  requirements,  particularly  those  unique  to  the  Government, 
usually  result  in  greater  risk  assumption  by  the  Government.  This  is 
especially  true  for  complex  research  and  development  contracts,  when 
performance  uncertainties  or  the  likelihood  of  changes  make  it  difficult  to 
estimate  performance  costs  in  advance. 

3.  ACQUISITION  PHASE  AND  PREFERRED  CONTRACT  TYPE 


As  a  program  progresses  through  the  acquisition  life  cycle  process,  risk  typically 
decreases.  During  the  concept  refinement  and  technical  development  phases,  risk  is  high. 
Critical  technologies  that  are  projected  to  mature  may  not  (INCOSE,  2004),  and  newer 
programs  may  be  more  susceptible  to  “paying  the  bills”  of  another  program  (Software 
Engineering  Institute  [SEI],  2009).  Approval  to  proceed  from  one  acquisition  phase  to 
another  comes  from  the  MDA,  as  we  explain  in  Chapter  III.  Figure  6  illustrates  the 
typical  contract  type  utilized  in  the  acquisition  process  by  life  cycle  phase. 
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Figure  6.  Contract  Type  by  Life  Cycle  Phase 

(DAU,  2008) 

Developmental  work  typically  requires  cost-reimbursement  contracts  (FAR, 


2011).  During  the  concept  refinement  and  technology  development  phases,  the 
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predominant  contract  type  is  CPFF  (DAU,  2008).  This  is  in  keeping  with  the  higher  risk 
associated  with  contract  performance  during  those  phases,  although  CPFF  is  not  the  only 
contract  type  used  before  MS  B.  R&D  continues  during  the  EMD  phase,  but  by  this 
point  programs  typically  demonstrate  enough  technical  maturity  and  design  stability  to 
permit  contract  types  with  less  government  cost  risk.  CPFF,  CPAF,  CPIF,  and  FFP 
contracts  are  common  during  the  EMD  phase  (DAU,  2008),  along  with  FPIF  contracts, 
which  we  examine  in  Chapter  V.  The  FAR  (2011)  recommends  that  when  follow-on 
production  requirements  have  been  contemplated  for  an  R&D  contract,  contracts  should 
progress  from  cost  reimbursement  to  fixed  price.  System  acquisition  contracts  during  the 
production  and  deployment  phase  tend  to  be  FPIF  or  FPAF,  and  sustainment  contracts  are 
often  FPIF,  FPAF,  or  FFP. 

Preferred  contract  types  shift  based  on  changing  policy  guidance,  phase,  or 
research.  A  study  (Sadeh,  Dvir,  &  Shenhar,  2000)  in  Israel  on  110  defense  development 
projects  found  that  cost-plus-incentive-fee  contracts  resulted  in  better  performance  when 
technological  uncertainty  was  high  at  the  start  of  the  project.  The  study  recommended  a 
combination  of  cost-plus  and  fixed-price  contracts  for  projects,  moving  toward  fixed 
price  when  uncertainty  is  reduced  (Sadeh  et  al.,  2000).  Debate  continues  as  to  when 
uncertainty  is  reasonably  low  enough  to  regularly  permit  fixed-price  contracts.  A  current 
DoD  proposed  rule  would  increase  the  use  of  FPIF  contracts  during  EMD  for  the  purpose 
of  attaining  better  contract  performance  (Levin,  J.,  2011).  In  later  chapters  of  this 
research  report,  we  examine  the  relative  performance  of  contracts  during  production 
years  on  the  basis  of  predominant  contract  type  during  EMD,  with  the  intent  of  solving 
the  debate  regarding  the  merit  of  various  contract  types  during  EMD. 

Because  the  contracting  officer  is  responsible  for  safeguarding  the  interests  of  the 
government  (FAR,  2011,  subpart  1.602-2),  the  ultimate  responsibility  to  appropriately 
select  a  program’s  contract  type  rests  with  him  or  her.  The  FAR  (2011)  states  that, 
“selecting  the  contract  type  is  generally  a  matter  for  negotiation  and  requires  the  exercise 
of  sound  judgment”  (subpart  16.103(a)).  With  the  exception  of  stating  that  the 
contracting  officer  is  responsible  for  determining  the  contract  type  through  negotiation 
with  contractors,  the  FAR  is  largely  silent  on  who  else  should  be  involved  in  the  decision, 
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with  the  exception  of  R&D  contracts.  In  practice,  determining  a  program’s  contract  type 
is  often  a  collaborative  decision  between  a  program’s  contracting  officer  and  program 
manager  or  contract  user  representative.  This  collaborative  approach  is  recommended  to 
contracting  officers  for  R&D  acquisitions  in  the  FAR  (2011)  due  to  “the  importance  of 
technical  considerations  in  R&D”  (subpart  35.006(b)).  DFARS  (2011)  subpart 
234.004(2)  goes  further  and  dictates  that  an  MDAP’s  MDA  should  select  the 
development  program  contract  type,  taking  into  consideration  the  contracting  officer’s 
recommendation,  at  the  time  of  MS  B  approval.  This  DFARS  exception  to  the  typical 
practice  of  the  contracting  officer  selecting  the  contract  type  reinforces  the  collaborative 
nature  of  determining  contract  type. 

The  FAR  (2011)  provides  “a  wide  selection  of  contract  types  ...  in  order  to 
provide  needed  flexibility  in  acquiring  the  large  variety  and  volume  of  supplies  and 
services  required  by  [the  government]”  (subpart  16.101).  The  flexibility  given  to 
contracting  officers  in  determining  contract  types  and  the  collaboration  suggested  in 
making  the  determination  of  contract  type  indicates  the  complex  nature  of  selecting  a 
contract  type,  regardless  of  the  acquisition  phase.  Accordingly,  no  particular  contract 
type  is  the  universal  answer  for  any  acquisition  phase. 

E.  IMPORTANCE  OF  RESEARCH  AND  DEVELOPMENT 

Four  main  causes  of  program  failure  have  been  identified  in  other  research, 
including  requirements  changes,  budget  instability,  technology  risk,  and  poor  execution 
due  to  inadequate  program  decision-making  information  (Miller,  2008).  R&D  reduces 
technology  risk  for  system  acquisition  programs  and  is,  thus,  critical  to  program  success 
both  during  and  after  the  main  acquisition  phases  associated  with  R&D,  which  are  as 
follows:  concept  refinement,  technology  development,  and  system  development  and 
demonstration.  Two  areas  that  contribute  to  the  relative  success  of  programs  during  and 
after  the  R&D  acquisition  phases  are  R&D  contracting  and  technology  readiness. 


1.  R&D  CONTRACTING 


Program  managers  must  use  careful  consideration  when  determining  the  type  of 


contract  used  for  R&D  efforts.  R&D  contracts  should  encourage  high  creativity  and 
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innovation.  The  FAR  (2011)  states  that  “the  primary  purpose  of  contracted  R&D 
programs  is  to  advance  scientific  and  technical  knowledge  and  apply  that  knowledge  to 
the  extent  necessary  in  order  to  achieve  agency  and  national  goals”  (subpart  35.002). 
Contracting  officers  must  judiciously  apply  the  FAR  policies  on  contract  type  to  achieve 
the  government’s  purpose  for  R&D  efforts.  MDAPs  are  only  responsible  for  a  portion  of 
R&D  spending  within  the  DoD.  Much  of  this  non-MDAP  spending  is  devoted  to  basic 
research,  including  technology  exploration  that  takes  place  long  before  technologies 
mature  to  the  point  necessary  to  begin  MDAPs.  The  primary  purpose  of  MDAP  R&D 
contracting  is  to  improve  the  technology  readiness  of  a  system’s  contemplated  critical 
technologies  through  developmental  and  applied  research. 

One  of  the  imperatives  of  R&D  contracting  is  to  establish  a  full  understanding 
between  the  parties  regarding  the  intent  of  the  R&D  effort  (FAR,  2011).  This 
understanding  can  be  impaired  due  to  misunderstanding  on  the  part  of  either  party  to  the 
contract,  resulting  in  R&D  efforts  that  do  not  meet  the  needs  of  the  government.  Due  to 
the  uncertainty  and  ambiguity  present  in  R&D  efforts,  the  FAR  (2011)  instructs 
contracting  officers  to  avoid  sealed  bidding.  Contracting  officers  should  instead  utilize 
competitive  negotiations  with  offerors  to  establish  a  comprehensive  mutual 
understanding  of  the  project  (FAR,  2011). 

Providing  flexibility  with  minimal  administrative  burden  should  be  considered  by 
contracting  officers  (FAR,  2011),  but  clear  objectives  must  be  conveyed  for  program 
success.  Contracting  officers  should  utilize  technical  personnel  to  define  clear  objectives 
for  R&D  projects  (FAR,  2011).  The  DoD  Space-Based  Infrared  System  (SBIRS)  is  an 
example  of  negative  results  due  to  unclear  objectives.  The  GAO  (2003)  reported  that  the 
cost  overruns  and  schedule  delays  for  SBIRS  began  at  the  inception  of  the  program  due 
to  immature  technologies  and  unclear  requirements.  In  keeping  with  the  FAR  (2011),  it 
is  critical  to  successful  developmental  and  applied  research  to  encourage  contractors  to 
“exercise  innovation  and  creativity”  in  responding  to  “clear  and  complete  ...  end 
objectives”  (subpart  35.005).  A  well-defined  research  objective,  combined  with  a  clear 
understanding  between  the  government  and  contractor,  contributes  to  the  relative  success 
of  a  program  by  removing  a  potential  impediment  to  MDAP  R&D  success. 
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2. 


TECHNOLOGY  READINESS  LEVELS 


The  level  of  uncertainty  during  R&D  varies  greatly,  particularly  during  the  early 
phases  of  a  program.  Improved  technology  readiness  is  theorized  to  contribute  to  the 
success  of  MDAPs  during  and  after  the  R&D  acquisition  phases.  Some  researchers 
suggest  that  the  relative  maturity  of  an  MDAP’s  technologies  contributes  to  improved 
cost  and  schedule  variance  outcomes. 

Some  programs  begin  R&D  with  more  mature  technologies,  but  others  have 
immature  technologies.  Program  technologies  must  advance  to  certain  minimum  levels 
before  programs  are  permitted  to  move  beyond  the  technology  development  and  EMD 
phases  (DoD,  2011).  Making  consistent,  informed  judgments  regarding  the  maturity  of 
an  MDAP’s  technology  requires  a  uniform  system  of  measurement.  One  method  of 
defining  the  maturity  of  a  program’s  technology  is  using  technology  readiness  levels 
(TRLs)  to  rate  the  readiness  of  a  technology  against  set  criteria.  Table  1  includes  a 
summary  of  the  TRL  descriptions  as  shown  in  section  10.5.2  of  the  Defense  Acquisition 
Guidebook  (DoD,  2011). 


Table  1.  TRL  Descriptions 

_ (DoD,  2011) _ 


Technology  Readiness  Level 

Description 

1 .  Basic  principles  observed 
and  reported. 

Lowest  level  of  technology  readiness.  Scientific  research  begins  to  be 
translated  into  applied  research  and  development.  Examples  might 
include  paper  studies  of  a  technology’s  basic  properties. 

2.  Technology  concept  and/or 
application  formulated. 

Invention  begins.  Once  basic  principles  are  observed,  practical 
applications  can  be  invented.  Applications  are  speculative  and  there  may 
be  no  proof  or  detailed  analysis  to  support  the  assumptions.  Examples 
are  limited  to  analytic  studies. 

3.  Analytical  and 
experimental  critical  function 
and/or  characteristic  proof  of 
concept. 

Active  research  and  development  is  initiated.  This  includes  analytical 
studies  and  laboratory  studies  to  physically  validate  analytical 
predictions  of  separate  elements  of  the  technology.  Examples  include 
components  that  are  not  yet  integrated  or  representative. 

4.  Component  and/or 
breadboard  validation  in 
laboratory  environment. 

Basic  technological  components  are  integrated  to  establish  that  they  will 
work  together.  This  is  relatively  “low  fidelity”  compared  to  the  eventual 
system.  Examples  include  integration  of  “ad  hoc”  hardware  in  the 
laboratory. 

5.  Component  and/or 
breadboard  validation  in 
relevant  environment. 

Fidelity  of  breadboard  technology  increases  significantly.  The  basic 
technological  components  are  integrated  with  reasonably  realistic 
supporting  elements  so  it  can  be  tested  in  a  simulated  environment. 
Examples  include  “high  fidelity”  laboratory  integration  of  components. 
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6.  System/subsystem  model  or 
prototype  demonstration  in  a 
relevant  environment. 

Representative  model  or  prototype  system,  which  is  well  beyond  that  of 
TRL  5,  is  tested  in  a  relevant  environment.  Represents  a  major  step  up  in 
a  technology’s  demonstrated  readiness.  Examples  include  testing  a 
prototype  in  a  high-fidelity  laboratory  environment  or  in  a  simulated 
operational  environment. 

7.  System  prototype 
demonstration  in  an 
operational  environment. 

Prototype  near,  or  at,  planned  operational  system.  Represents  a  major 
step  up  from  TRL  6,  requiring  demonstration  of  an  actual  system 
prototype  in  an  operational  environment  such  as  an  aircraft,  vehicle,  or 
space.  Examples  include  testing  the  prototype  in  a  test  bed  aircraft. 

8.  Actual  system  completed 
and  qualified  through  test  and 
demonstration. 

Technology  has  been  proven  to  work  in  its  final  form  and  under  expected 
conditions.  In  almost  all  cases,  this  TRL  represents  the  end  of  true 
system  development.  Examples  include  developmental  test  and 
evaluation  of  the  system  in  its  intended  weapon  system  to  determine  if  it 
meets  design  specifications. 

9.  Actual  system  proven 
through  successful  mission 
operations. 

Actual  application  of  the  technology  in  its  final  form  and  under  mission 
conditions,  such  as  those  encountered  in  operational  test  and  evaluation. 
Examples  include  using  the  system  under  operational  mission  conditions. 

The  use  of  TRLs  is  credited  with  enabling  “consistent,  uniform,  discussions  of 


technical  maturity  across  different  types  of  technologies”  (DoD,  201 1,  p.  853).  TRLs  are 
limited,  however,  to  maturity  and  do  not  directly  address  the  probability  that  a  part  or 
subsystem  will  achieve  the  maturity  required  for  system  production.  TRLs  are  also 
assessed  on  an  ordinal  scale,  so  it  is  incorrect  to  assume  that  a  system  will  require 
equivalent  effort  to  increase  from  one  level  to  the  next.  Current  DoD  policy  requires 
MDAPs  to  achieve  TRL  6  by  MS  B  and  TRL  7  by  MS  C  (DoD,  2011).  The  Defense 
Acquisition  Guidebook  (DoD,  2011)  also  instructs  MDAs  to  “consider  the  recommended 
TRLs  . . .  when  assessing  program  risk.”  However,  TRLs  should  not  be  the  sole  measure 
of  a  program’s  technical  risk  because  they  do  not  measure  the  potential  impact  of  failing 
to  achieve  technology  maturity.  TRLs  cannot  offer  a  full  assessment  of  program  risk,  but 
they  do  provide  decision-makers  with  uniform,  comprehensible  data  regarding  a 
program’s  present  technology  maturity. 


Miller  (2008)  blamed  requirements  changes,  technology  risk,  and  poor  execution 
due  to  insufficient  or  improper  program  decision-making  information  as  three  of  the 
causes  of  poor  performance  for  MDAPs.  Programs  entering  the  technology  development 
phase  with  higher  minimum  and  average  TRLs  have  been  shown  to  exhibit  lower  cost 
growth  and  schedule  growth  (Dillard  &  Ford,  2009).  Higher  levels  of  technology 
readiness  could  operate  in  an  inverse  relationship  to  technology  risk:  an  MDAP  with 
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increased  TRLs  relative  to  its  acquisition  phase  would  then  exhibit  lower  technology  risk. 
Increasing  the  impartial  use  of  TRLs  could  also  improve  program  decision-making 
(GAO,  1999).  It  is  therefore  possible  that  higher  TRLs  relative  to  an  MDAP’s 
acquisition  phase  could  possibly  mitigate  Miller’s  (2008)  causes  of  poor  performance. 

Improved  technology  readiness  contributes  to  the  success  of  programs  during  and 
after  the  R&D  acquisition  phase.  Recent  evolutionary  acquisition  efforts  within  the  DoD 
have  focused  on  using  more  mature  technologies  to  develop  incremental  capabilities. 
Some  researchers  suggest  that  programs  with  higher  TRLs  tend  to  exhibit  better  cost  and 
schedule  performance  (Dillard  &  Ford,  2009).  Differences  in  TRLs  may  explain  some  of 
the  variation  not  otherwise  explained  by  the  variables  we  model  in  later  chapters. 

F.  SELECTED  ACQUISTION  REPORTS 

Selected  Acquisition  Reports  (SARs),  which  are  published  annually,  are  one  of 
the  main  sources  of  data  we  used  for  the  analysis  in  this  study.  These  reports  provide  a 
snapshot  in  time  for  each  program  we  analyzed.  As  mandated  by  Congress  under  10 
U.S.C.  §2432  (2011),  the  Secretary  of  Defense  must  submit  SARs  for  all  MDAPs  or 
programs  designated  as  high  interest  by  the  USD(AT&L).  Congress  utilizes  these  reports 
to  track  the  progress  of  MDAPs,  specifically  to  detect  early  warnings  of  cost  or  schedule 
overruns  (DoD,  2011). 

SARs  summarize  the  latest  status  of  total  program  cost,  schedule,  and 
performance  as  well  as  program  unit  cost  and  unit  cost  breach  information.  Each  SAR 
provides  a  full  life  cycle  cost  analysis  for  the  reporting  program  and  is  prepared  annually 
in  conjunction  with  submission  of  the  president’s  budget.  Subsequent  quarterly 
exception  reports  are  required  only  for  those  programs  experiencing  unit  cost  increases  of 
15%  or  more,  or  schedule  delays  of  six  months  or  greater  since  the  current  estimate 
reported  in  the  previous  SAR,  or  when  MS  B  or  MS  C  approval  occurs  within  the 
reportable  quarter  (DoD,  2011). 
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G.  DEFENSE  ACQUISITION  EXECUTIVE  SUMMARIES 


Defense  Acquisition  Executive  Summary  (DAES)  reports  are  published  quarterly 
and  are  the  other  source  of  data  for  the  analysis  we  conducted  in  this  research.  DAES 
reports  provide  a  comprehensive  summary  of  AC  AT  I  and  AC  AT  1A  programs  between 
milestone  decision  points.  The  DAES  reports  must  contain  program  assessments,  unit 
costs,  current  estimates  of  program  baselines,  and  the  status  of  exit  criteria.  They  present 
the  projected  total  costs  and  quantities  for  all  remaining  years  of  an  acquisition  program’s 
life.  DAES  information  is  designed  to  provide  indications  of  both  potential  and  actual 
program  problems  to  the  USD(AT&L)  and  to  the  Assistant  Secretary  of  Defense 
(Networks  &  Information  Integration)  before  they  become  significant.  The  reports 
provide  action  taken  or  planned  to  mitigate  future  program  problems  (DoD,  2011). 

H.  SUMMARY 

In  this  chapter,  we  summarized  the  background  information  required  to 
understand  the  scope  of  this  research.  We  presented  the  three  support  systems  that  form 
the  DoD’s  acquisition  system,  and  we  briefly  explained  the  federal  contract  types 
applicable  to  commercial  acquisitions  and  the  preferred  contract  types  for  each  DoD 
acquisition  phase.  We  concluded  the  chapter  with  the  importance  of  R&D  to  acquisition 
program  success  and  discussed  the  primary  documents  from  which  this  study  draws. 
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III.  PURPOSE 


A.  OVERVIEW 

The  purpose  of  this  research  is  to  better  understand  indicators  of  program  and 
contract  cost  variances.  Establishing  accurate  cost  information  prior  to  entering  the  EMD 
and  production  phases  is  extremely  important  to  ensuring  a  successful  program  (Carter, 
2011).  We  propose  that  preproduction  and  contract-level  variances,  including  those 
reported  during  the  EMD  phase,  are  one  possible  set  of  indicators  of  future  program  cost 
variances.  Interpreting  these  variances  requires  an  understanding  of  the  reasoning  behind 
them  and  of  their  methods  of  construction. 

In  this  chapter,  we  provide  the  information  required  to  understand  program-  and 
contract-level  variances,  including  program  cost,  schedule,  and  engineering  variances,  as 
well  as  contract  earned  value  cost  and  schedule  variances.  Previous  research  has  been 
performed  on  cost  overruns  by  contract  type  (Berteau  et  al.,  2011),  and  recent  initiatives 
have  recommended  the  increased  use  of  fixed-price  contracts  (Carter,  2010).  One 
purpose  of  this  research  is  to  identify  the  effect  of  fixed-price  contracts  on  program 
variances. 

In  this  chapter,  we  explain  the  regulations  that  govern  these  variances  and  that 
inform  the  decision-making  of  acquisition  professionals.  We  also  define  each  of  the 
program  and  contract  variances  referenced  in  this  research.  We  conclude  the  chapter 
with  an  explanation  of  why  the  study  of  these  acquisition  variances  is  important. 

B.  ACQUISITION  OVERSIGHT 

The  defense  acquisition  process  includes  extensive  program  oversight  and  data 
reporting  requirements.  Program  oversight  has  not  produced  the  desired  effect — capable 
systems  delivered  on  time  and  within  budget.  However,  a  closer  review  of  the  program 
and  contract  variance  data  collected  under  present  reporting  requirements  may  reveal 
novel  conclusions  about  the  interdependent  nature  of  the  problems  associated  with 
defense  acquisition. 
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The  current  system  of  acquisition  oversight  is  the  result  of  numerous  reforms 
(Rothenflue  &  Kwolek,  2010).  In  the  following  sections,  we  discuss  some  of  these 
reforms,  including  the  Nunn-McCurdy  Amendment  and  several  other  influential  reforms, 
which  substantially  contributed  to  the  structure  and  content  of  the  variance  data  analyzed 
in  this  research. 

1.  The  Nunn-McCurdy  Amendment 

One  reform  that  significantly  changed  oversight  procedures  and  focused  on 
controlling  cost  growth  was  the  Nunn-McCurdy  Amendment.  Nunn-McCurdy,  first 
introduced  in  the  Department  of  Defense  Authorization  Act  of  1982,  expanded  the  SAR 
requirements  established  in  1969  (Rothenflue  &  Kwolek,  2010).  Nunn-McCurdy 
required  program  managers  to  submit  SARs  to  Congress  annually  or  immediately 
following  a  growth  of  15%  over  the  total  program  acquisition  unit  cost  (PAUC)  and 
average  procurement  unit  cost  (APUC).  Nunn-McCurdy  is  still  a  public  law,  so  program 
managers  are  still  required  to  submit  SARs.  SARs  must  include  any  change  in  schedule 
milestones  and  system  performance  (Department  of  Defense  Authorization  Act,  1982). 
Additional  reporting  guidance  has  been  added  to  the  Nunn-McCurdy  statute  over  the 
years  and  that  guidance  remains  today  as  a  control  method  for  holding  program  managers 
accountable  for  cost  growth  on  MDAPs. 

The  most  recent  changes  to  the  statute  occurred  in  2006  and  2009.  The  National 
Defense  Authorization  Act  for  Fiscal  Year  2006  amended  Nunn-McCurdy  to  include 
significant  and  critical  dollar  amount  thresholds  rather  than  a  single  threshold  (GAO, 
2011a).  In  the  act,  significant  cost  growth  is  defined  as  a  15%  increase  to  the  current 
baseline  or  a  30%  increase  to  the  original  baseline  for  the  PAUC  or  the  APUC.1 
Significant  cost  growth  requires  congressional  reporting.  Critical  cost  growth  is  defined 
as  a  25%  increase  to  the  current  baseline  estimate  or  a  50%  increase  over  the  original 
baseline  estimate  for  the  PAUC  or  APUC  (GAO,  2011a). 


1  PAUC  =  (Total  Development  $  +  Procurement  $  +  Construction  $)/Total  Program  Quantity.  APUC  = 
Total  Procurement  S/Procurement  Quantity. 
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It  is  important  to  understand  estimates  and  baselines  when  discussing  the  Nunn- 
McCurdy  Amendment.  The  current  baseline  estimate  is  defined  as  the  latest  estimate  on 
an  approved  program,  defined  as  the  currently  approved  acquisition  program  baseline 
(APB).  The  original  baseline  estimate  is  defined  as  the  APB  approved  at  MS  B  or 
program  initiation,  whichever  occurs  later  (Axtell  &  Irby,  2007).  Each  military  Service 
must  establish  a  baseline  for  each  of  its  MDAPs,  including  parameters  to  describe  the 
cost  estimate  (referred  to  as  the  baseline  estimate),  schedule  estimate,  performance 
estimate,  and  any  other  important  factors  of  an  MDAP  (10  U.S.C.  §  2435,  2011). 
According  to  10  U.S.C.  §  2435  (2011),  the  revision  of  the  original  baseline  should  be 
changed  to  the  new  baseline  only  after  a  critical  (Nunn-McCurdy)  breach;  if  this  happens, 
the  Secretary  of  Defense  notifies  Congress  of  the  breach  in  the  next  SAR  and  gives 
reasons  for  the  adjustment  or  revision  (10  U.S.C.  §  2435,  2011). 

The  Weapon  Systems  Acquisition  Reform  Act  (WSARA)  of  2009  added  further 
repercussions  for  programs  with  critical  cost  growth.  Among  other  requirements, 
WSARA  added  the  presumption  that  a  program  will  be  cancelled  unless  the  Secretary  of 
Defense  “certifies  (with  reasons  and  supporting  documentation)  that  continuing  the 
program  is  essential  to  national  security  and  that  the  program  can  be  modified  to  proceed 
in  a  cost-effective  manner”  (Lymon,  McWhorter,  &  Violette,  2011,  p.  20).  Critical  cost 
growth  also  requires  the  program  to  “receive  a  new  milestone  approval  (and  associated 
certification)  prior  to  the  award  of  any  new  contract  or  contract  modification  extending 
the  scope  of  the  program”  (Lymon  et  al.,  201 1,  p.  20). 

WSARA  also  requires  departments  to  perform  an  independent  cost  estimate 
supporting  a  program’s  cost  reasonableness  and  a  stated  confidence  level  for  that  estimate 
(Levin,  C.,  2009).  The  requirement  for  a  stated  confidence  level  echoes  an  observation 
made  by  the  GAO  (2008)  in  a  report  to  the  Senate  Armed  Service  Committee:  “To  make 
more  informed  investment  decisions,  cost  estimating  best  practices  call  for  estimating  a 
range  of  possible  costs  around  a  point  estimate  to  provide  information  about  the  levels  of 
uncertainty  and  confidence”  (p.  24).  There  were  seven  Nunn-McCurdy  breaches  in  2010 
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to  the  APBs — four  significant  and  three  critical  (USD[AT&L],  2011).  In  the  following 
section,  we  explain  APBs,  the  initial  program  baselines  that  Nunn-McCurdy  breaches 
were  based  on. 

2.  Acquisition  Program  Baseline 

The  APB  was  established  by  the  Defense  Acquisition  Improvement  Act  of  1986 
to  create  a  baseline  to  improve  program  stability.  The  APB  is  defined  by  the  DAU  as  the 
“baseline  that  reflects  the  threshold  and  objective  values  for  the  minimum  number  of 
cost,  schedule,  and  performance  attributes  (called  ‘key  performance  parameters’)  that 
describe  the  program  over  its  life  cycle”  (“Acquisition  Program  Baseline,”  2009).  The 
APB  answers  how  the  system  is  supposed  to  perform  when  critical  events  occur,  and  how 
much  the  program  should  cost.  Every  program  manager  must  submit  and  receive 
approval  for  program  goals  prior  to  initiation  of  all  acquisitions  (USD[AT&L],  2008). 
The  APB  satisfies  the  requirement  for  goals  on  all  ACAT  I  programs. 

By  tracking  actual  program  performance  against  established  baselines,  the 
program  manager  is  alerted  to  potential  problems  and  can  take  early  corrective  action.  If 
a  program  breaches  an  approved  baseline  threshold,  the  program  manager  must  submit  a 
formal  memo  to  the  MDA  and  to  the  component’s  leadership.  A  breach  of  performance 
is  defined  as  a  failure  to  meet  the  specific  parameter’s  threshold  value  as  laid  out  in  the 
APB.  A  breach  of  schedule  is  failure  to  meet  the  objective  date  plus  six  months.  A 
cumulative  program  cost  increase  of  10%  or  greater  from  the  approved  cost  baseline  is  a 
cost  breach  (“Acquisition  Program  Baseline,”  2009). 

3.  Selected  Acquisition  Reports 

SAR  requirements  are  covered  under  10  U.S.C.  §  2432  (201 1).  SAR  submittal  is 
required  annually  on  December  3 1  unless  a  significant  or  critical  cost  growth  occurred  or 
a  delay  of  six  months  in  any  current  estimated  milestone  occurs,  both  of  which  require  a 
quarterly  report  (10  U.S.C.  §  2432,  2011).  SARs  contain  the  following  19  sections: 

1 .  Program  Identification, 

2.  Program  Information, 
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3.  Responsible  Office, 

4.  References, 

5.  Mission  and  Description, 

6.  Executive  Summary, 

7.  Threshold  Breaches, 

8.  Schedule, 

9.  Performance, 

10.  Track  to  Budget, 

1 1 .  Cost  and  Funding, 

12.  Low-Rate  Initial  Production, 

13.  Nuclear  Cost, 

14.  Foreign  Military  Sales, 

15.  Unit  Cost, 

16.  Cost  Variance, 

17.  Contracts, 

18.  Deliveries  and  Expenditures,  and 

19.  Operating  and  Support  Cost. 

Section  16,  Cost  Variance,  is  reported  in  a  SAR  in  the  following  four  steps.  In  the 
first  step,  the  total  variance  for  each  program’s  appropriations  estimate  is  calculated. 
Second,  the  category  for  the  particular  change  is  identified.  Third,  the  dollar  amount  for 
each  variance  category  by  FY  is  determined.  Finally,  a  clear  and  understandable 
explanation  for  the  changes  is  provided  (DoD,  2010). 

A  program’s  total  CV  comes  from  many  different  sources.  These  are  aggregated 
in  SARs  into  seven  categories.  In  later  chapters  of  this  research  report,  we  describe  the 
relationships  deduced  from  SAR  CV  data  and  other  data.  Accordingly,  it  is  important  to 
understand  the  categories  comprising  the  program’s  total  CV,  which  are  shown  in  Figure 
7. 
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Figure  7.  Total  Program  Variance  Factors 

(DoD,  2010) 

The  various  cost  variances  are  calculated  for  the  SARs,  which  are  uploaded  to  DAMIRS. 


The  variance  that  occurred  between  reports  for  each  factor  in  Figure  7  must  be  verbally 


explained  in  the  SAR. 


According  to  a  GAO  (2011a)  study,  engineering  and  design  issues,  schedule 
issues,  and  quantity  changes  were  the  primary  reasons  cited  for  unit  cost  growth  that  led 
to  Nunn-McCurdy  breaches.  In  a  study  performed  by  David  McNicol  (2004)  on  cost 
growth,  he  identified  three  areas  that  caused  growth  in  procurement  costs:  increased 
system  capabilities,  an  unrealistic  estimate  of  cost  growth,  and  poor  program  execution  or 
exceptional  budget  instability.  In  this  research,  we  focus  on  CV,  SV,  and  EV  as  potential 
leading  indicators  of  relative  program  success  or  failure  since  the  other  primary  reason — 
quantity  change — is  more  likely  to  be  a  lagging  rather  than  leading  indicator  (Gansler, 
2010). 
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4.  Earned  Value  Management 

Former  USD(AT&L)  Dr.  Jack  Gansler  signed  a  memorandum  in  August  1999 
announcing  the  DoD’s  adoption  of  the  Earned  Value  Management  (EVM)  system  (Office 
of  the  Secretary  of  Defense,  2002).  EVM  is  a  mandatory  reporting  requirement  for  cost 
and  incentive  contracts  valued  at  $20  million  or  greater,  to  include  all  MDAPs,  and  is 
governed  by  DFARS  (2011)  subpart  234.2  and  DoD  Instruction  5000.02  (USD[AT&L], 
2008).  The  use  of  EVM  is  optional  for  the  program  manager  on  any  contract  valued  at 
less  than  $20  million  based  on  a  cost-benefit  analysis  (USD[AT&L],  2008). 

EVM  is  a  tool  used  to  manage  programs  that  attempts  to  integrate  contract  cost, 
schedule,  and  technical  parameters  in  order  to  hold  all  parties  accountable  for  large, 
complex  acquisitions.  Program  managers  are  ultimately  accountable  for  confirming  that 
EVM  requirements  are  included  in  statements  of  work,  and  program  managers  can  utilize 
the  variance  measurements  to  forecast  contract  cost  and  schedule  performance.  The 
program  manager  can  use  EVM  to  track  the  status  of  the  contracts  within  his  or  her 
program  and  use  the  measurements  from  EVM  to  build  corrective  action  plans  to  get  the 
program  back  on  track. 

Problems  with  individual  contracts  may  impede  the  progress  of  a  portion  of  a 
program,  or  they  may  cause  cascading  problems  across  multiple  contracts  within  a 
program  and  the  program  as  a  whole.  In  later  chapters,  we  examine  contract  cost  and 
schedule  variances  as  determinants  of  relative  program  success.  In  the  following  section, 
we  explain  the  main  SAR  output  of  EVM,  contract  cost  and  schedule  variances,  in  greater 
detail.  For  the  sake  of  clarity,  in  this  research  we  avoid  referring  to  earned  value  as  “EV” 
in  order  to  better  distinguish  earned  value  from  engineering  cost  variance  (EV),  which  we 
define  later  in  this  section. 

C.  CONTRACT  VARIANCES 

Two  main  EVM  metrics — cost  variance  and  schedule  variance —  are  reported  for 
every  active  MDAP  contract,  meaning  every  MDAP  contract  that  is  less  than  90% 
complete.  Multiple  cost  and  schedule  contract  variances  are  routinely  reported  on  a 
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single  program  SAR.  To  better  understand  contract  cost  variance  and  schedule  variance, 
we  further  define  these  terms  in  the  next  sections. 

1.  Contract  Cost  Variance 

Contract  cost  variance  is  a  metric  that  calculates  cost  performance  by  subtracting 
the  actual  cost  of  work  performed  (ACWP)  from  the  budgeted  cost  of  work  performed 
(BCWP).  The  value  is  at  a  particular  point  of  time  and  shows  cumulative  cost  variance  to 
date.  A  positive  value  is  favorable  because  it  indicates  the  work  was  performed  under 
budget.  A  negative  value  is  unfavorable  because  it  indicates  more  money  was  spent  than 
was  budgeted  for  the  task. 

Contracts  experiencing  unfavorable  cost  variance  are  likely  to  experience  a 
contract  cost  overrun  because  it  is  difficult  to  reduce  budgeted  future  work  unless 
quantities  or  scope  of  work  are  reduced  (“Cost  Variance,”  2009).  SAR  contract  cost 
variances  are  reported  as  the  change  in  BCWP  minus  ACWP  rather  than  in  current-year 
or  base-year  dollars.  For  the  sake  of  clarity,  in  this  research  we  avoid  abbreviating 
contract  cost  variance  as  “CV”  in  order  to  better  distinguish  contract  cost  variance  from 
program  cost  variance  (CV),  which  we  define  later  in  this  section. 

2.  Contract  Schedule  Variance 

Contract  schedule  variance  is  a  metric  that  calculates  schedule  performance  by 
subtracting  the  budgeted  cost  of  work  scheduled  (BCWS)  from  the  BCWP.  The  value  is 
expressed  for  a  specific  period  of  time  or  is  cumulative  to  date.  A  positive  value  is 
favorable  because  it  indicates  more  work  than  scheduled  has  been  completed.  A  negative 
value  is  unfavorable  because  it  indicates  that  planned  work  was  not  completed.  Both  a 
slip  in  schedule  and  a  failure  to  achieve  certain  technical  milestones  as  planned  can  result 
in  a  negative  schedule  variance. 

Programs  experiencing  unfavorable  schedule  variance  may  also  experience  a 
delayed  completion,  but  they  can  possibly  recover  in  the  future  (“Schedule  Variance,” 
2009).  For  the  sake  of  clarity,  in  this  research  we  avoid  abbreviating  contract  schedule 
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variance  as  “SV”  in  order  to  better  distinguish  contract  schedule  variance  from  program 
schedule  variance  (SV),  which  we  define  later  in  this  section. 

D.  PROGRAM  VARIANCES 

Over  the  last  20  years,  the  Army  cancelled  22  major  programs  before  they  entered 
production;  the  costs  already  incurred  were  approximately  $1  billion  in  1996  and  reached 
a  high  of  $3.8  billion  per  year  after  2004  (Ewing,  2011).  Common  causes  for  program 
cancellation  are  cost  overruns,  schedule  slips,  and  underestimation  or  failure  to  eliminate 
technological  risk.  In  our  research,  we  examine  at  the  program  level  three  corresponding 
categories  of  variance  (cost,  schedule,  and  engineering  variance)  found  in  the  SARs.  To 
better  understand  the  difference  between  these  program  variances,  we  further  define  them 
in  the  next  sections. 

1.  Cost  Variance 

Program  cost  variance  (CV)  is  reported  in  two  general  forms:  as  the  cost  change 
between  the  program’s  current  SAR  and  previous  SAR,  which  is  the  indicator  we 
employ.  In  addition  the  cost  change  between  the  program’s  current  SAR  and  SAR 
baseline  is  also  reported.  A  positive  value  is  unfavorable  because  it  indicates  that  the 
estimated  cost  of  the  program  has  increased.  A  negative  value  is  favorable  because  it 
indicates  that  the  estimated  cost  of  the  program  has  decreased.  Changes  in  CV  are 
subdivided  in  seven  categories:  economic,  quantity,  schedule,  engineering,  estimating, 
other,  and  support.  The  variances  associated  with  these  seven  categories  sum  to  the  cost 
variance.  We  examine  two  of  these  categories,  schedule  and  engineering,  more  closely 
in  the  following  sections. 

2.  Schedule  Variance 

Schedule  variance  (SV)  is  cost  variance  attributable  to  schedule  changes.  SV  is  a 
component  of  CV.  Since  SV  is  a  component  of  CV,  the  two  will  covary.  Accordingly, 
the  CV  we  used  in  this  research  provides  explicit  explanatory  information  about  the 
subcomponents  SV  and  EV. 
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3.  Engineering  Variance 

Engineering  variance  (EV)  is  cost  variance  attributable  to  engineering  changes 
and  is  a  component  of  program  cost  variance.  EV  is  more  difficult  to  quantify  than  SV 
and  is  referred  to  in  DoD  contracting  offices  as  the  cost  of  an  approved  engineering 
change  proposal.  This  variance  occurs  due  to  new  technology  upgrades,  redesign,  and 
configuration  changes.  As  with  all  SAR  cost  variance  categories,  a  descriptive 
explanation  is  provided  in  the  SAR  to  aid  understanding  of  what  caused  the  variance. 
Changes  in  support  items  are  not  included  in  EV  (DoD,  2010). 

Engineering  changes  typically  occur  to  a  specific  item  identified  on  the  work 
breakdown  structure  (WBS).  A  WBS  defines  the  deliverable  element  by  the  scope  of 
work.  If  the  scope  changes,  an  engineering  variance  to  total  cost  will  likely  occur. 
Minimizing  changes  and  beginning  a  program  with  a  clear  and  well-defined  requirement 
is  the  best  prevention  for  poor  engineering  variance  performance  (GAO,  2011). 

Because  engineering  cost  variance  is  a  component  of  total  program  cost  variance, 
the  presence  of  any  engineering  cost  variance  will  cause  the  two  to  display  covariance. 

As  a  result,  when  we  discuss  CV,  it  encompasses  SV  and  EV.  Because  of  this,  when 
conducting  much  of  the  analysis  on  the  models  we  developed  in  this  paper,  we  subtracted 
SV  and  EV  from  CV.  We  call  this  Program  Net  Cost  Variance  later  in  the  analysis. 

E.  REASONS  TO  STUDY 

Understanding  what  variables  affect  program  costs  and  CVs,  and  the  linkages 
between  contract  and  program  data,  is  the  purpose  of  this  research.  To  better  understand 
possible  drivers  of  program  CVs,  we  examine  the  historical  effects  of  preproduction 
decisions  on  production  CV  outcomes.  Improving  program  cost  outcomes  has  been  the 
subject  of  many  acquisition  reforms,  but  these  reforms  have  not  addressed  the  complexity 
and  interdependence  found  in  DoD  acquisition  programs  that  drive  poor  program  cost, 
schedule,  and  technical  outcomes  (Rothenflue  &  Kwolek,  2010). 

We  examine  several  potential  cost  variance  drivers  in  later  chapters,  including 
contract  type  during  the  EMD  acquisition  phase,  program  and  contract  variances,  and 
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MDAP  type.  It  would  be  beneficial  to  the  acquisition  workforce  to  better  understand 
each  of  these  potential  causes  of  variance. 


1.  R&D’s  Effect  on  Life  Cycle  Costs 

R&D  contracts  for  FY2010  totaled  $80  billion  (USD  Comptroller,  2011). 
Increasing  the  number  of  R&D  contracts  issued  on  a  fixed-price  incentive  basis  could 
yield  significant  savings  based  on  the  presumption  that  fixed-price  incentive  contracts 
decrease  cost  overruns  (Kendall,  2011).  At  one  time,  the  DoD  attempted  to  impose 
fixed-price  incentive  contracts  on  efforts  in  which  significant  invention  could  be 
anticipated,  although  recently  the  use  of  cost-plus-award-fee  contracts  has  become 
widespread  (Carter,  2010;  Darst  &  Roberts,  2010).  Under  Secretary  Carter  (2010)  has 
advocated  the  use  of  fixed-price  incentive  firm  target  contracts  in  the  place  of  cost-plus- 
award-fee  contracts  wherever  practicable,  including  EMD  contracts  which  are  of 
particular  interest  to  policy-makers  (Carter,  2011). 

The  changing  political  favorability  of  particular  contract  types  must  be  evaluated 
with  the  regulations  implicit  in  the  13  federal  contract  types  listed  in  FAR  (201 1)  part  16; 
different  contract  types  are  better  suited  to  different  types  of  work.  This  truth  is 
consistent  with  the  discretion  given  to  the  acquisition  team  in  FAR  (2011)  subpart  1.102 
to  “use  sound  business  judgment.”  The  contract  types  most  significant  to  MDAPs  were 
discussed  in  Chapter  II. 

Program  managers  and  contracting  officers  should  understand  that  the  selection  of 
an  inappropriate  contract  type  during  EMD  can  negatively  impact  contract  performance 
(Sadeh  et  al.,  2000).  Restricting  R&D  work  to  a  fixed-price  incentive  basis  could 
negatively  affect  problem  detection  and  problem  solving  early  in  a  program  if  the  risk 
conditions  are  such  that  a  fixed-price  contract  type  is  not  appropriate.  Contracted 
companies  could  avoid  researching  all  problems  and  alternative  program  solutions  in 
order  to  increase  their  profit  margin  on  the  fixed-price  contracts,  provided  that  the 
contract  specification  requirements  are  satisfied.  Poor  problem  detection  early  in  a 
program  will  likely  increase  program  costs  in  the  long-term.  Good  EMD  contract 
performance  and  appropriate  problem  solving  can  address  potential  technical  and  system 
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integration  issues  before  production,  when  they  are  most  costly  to  address.  Studying  the 
effect  of  EMD  contract  type  may  give  important  insight  into  the  wisdom  of  incentivizing 
program  managers  to  select  more  restrictive  contract  types. 

2.  Aid  Current  Practitioners’  Programmatic  Decision-Making 

Examining  potential  sources  of  variance  may  improve  program  managers’ 
understanding  of  reasons  for  the  relative  success  of  programs.  A  better  understanding  of 
these  reasons  could  aid  the  decision-making  of  current  practitioners.  Preproduction 
program  cost,  schedule,  and  engineering  variance,  and  current  period  contract  earned 
value  cost  and  schedule  variance  are  proposed  by  acquisition  managers  as  possible 
indicators  of  future  program  cost  variances.  CV,  SV,  and  EV  within  individual  programs 
have  been  qualitatively  linked  and,  to  an  extent,  quantitatively  linked  (Rothenflue  & 
Kwolek,  2010).  Further  study  is  necessary  to  better  determine  the  nature  of  the 
interdependence  between  these  program  and  contract  variances.  An  improved 
understanding  of  this  interdependence  could  enhance  practitioners’  ability  to  estimate  a 
contract’s  most  likely  final  cost,  a  program’s  schedule  or  cost  variances,  or  a  program’s 
relative  cost  risk,  given  past  program  and  contract  variances. 

Other  factors  are  proposed  by  acquisition  managers  as  possible  reasons  for 
program  cost  variances,  including  MDAP  segment.  An  MDAP  segment  is  the  acquisition 
portfolio  segment  that  a  program  falls  into,  such  as  aircraft,  missiles,  ammunition, 
shipbuilding,  or  other,  as  defined  by  the  categories  listed  in  the  National  Defense  Budget 
Estimates  for  FY2012  (USD  Comptroller,  2011).  In  this  research,  we  do  not  further 
differentiate  between  these  categories,  although  such  differentiations  can  be  found  in  the 
Air  Force’s  Appropriation  Symbols  and  Budget  Codes  (Fiscal  Year  2012;  Department  of 
the  Air  Force,  2011),  in  order  to  permit  program  comparisons  between  similar  programs 
across  Services.  Some  studies  of  cost  and  schedule  variances  have  been  restricted  to  a 
particular  MDAP  segment,  such  as  aircraft  (Rothenflue  &  Kwolek,  2010);  such  studies 
inherently  exclude  the  possible  effects  of  the  MDAP  segment.  Constraining  a  study  to  a 
particular  contract  type  or  MDAP  segment  can  control  for  effects  associated  with  details 
of  that  specific  type  of  contract.  Although  it  is  unlikely  that  policy-makers  would 
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develop  fewer  programs  of  a  given  MDAP  segment  simply  due  to  the  relative  success  of 
that  type  of  program,  examining  the  relative  success  of  the  various  MDAP  categories  in 
conjunction  with  other  factors  could  improve  acquisition  professionals’  understanding  of 
the  relative  risk  present  in  each  MDAP  segment. 

Improving  practitioners’  understanding  of  the  risks  present  in  MDAPs  based  on 
past  program  performance  may  enhance  future  program  outcomes  by  aiding  cost 
estimating  and  improving  program  risk  assessment.  Such  developments  could  improve 
program  outcomes  by  reducing  funding  volatility  due  to  cost  overruns  and  could  decrease 
technical  risk  by  improving  risk  assessment.  Other  improvements  are  possible,  including 
program  improvements  driven  by  further  reform  of  the  acquisition  system  such  as  setting 
shorter  program  timelines,  promoting  real  competition,  and  increasing  the  use  of  fixed- 
price  incentive  contracts  (Carter,  2010). 

3.  Recommended  Future  R&D  Management  Reforms 

A  better  understanding  of  cost  and  schedule  variances  could  identify  areas  of 
current  policy  that  would  benefit  from  reform.  Novel  findings  regarding  any  of  the 
researched  causes  of  program  variances  could  yield  potential  reform  improvements. 
Conversely,  research  conducted  on  the  effects  of  acquisition  reforms  on  defense 
programs  could  provide  direction  for  creating  more  effective  policies  (Searle,  1997). 

Relating  the  efficacy  of  past  reforms  through  a  review  of  various  yearly  effects  on 
variances  over  time  could  permit  a  qualitative  review  of  past  reform.  Two  such  studies 
(Christensen,  Searle,  &  Vickery,  1999;  Drezner,  Jarvaise,  Hess,  Hough,  &  Norton,  1993) 
showed  that  program  reform  initiatives  did  not  improve  program  performance  and,  in 
fact,  cost  growth  worsened  after  initiatives  were  implemented.  It  could  be  useful  to 
question  the  conventional  thinking  that  past  reforms,  such  as  the  Federal  Acquisition 
Streamlining  Act  (FAS A)  of  1994,  have  had  a  negligible  effect  on  MDAP  outcomes 
(Smirnoff  &  Hicks,  2008).  The  most  recent  acquisition  reforms,  such  as  WSARA,  have 
generally  focused  on  reducing  risk.  It  is  possible  that  such  reforms  may  already  be 
improving  the  acquisition  process. 
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F. 


SUMMARY 


The  purpose  of  this  research  is  to  better  understand  the  determinants  of  program 
and  contract  cost  and  cost  variances.  Each  possible  cause  of  program  variance  discussed 
in  this  chapter  is  worthy  of  further  research  in  order  to  understand  the  possible  causes  of 
program  cost  variances.  The  various  program  cost  variances  and  contract  variances  we 
discussed  in  this  chapter  can  be  qualitatively  linked  to  cost,  schedule,  and  technical 
performance.  We  explore  these  interdependent  variables  and  other  possible  causes  of 
variance  quantitatively  in  Chapter  IV. 
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IV.  DATABASE  CONSTRUCTION  AND  ANALYSIS 


A.  OVERVIEW 

In  this  chapter,  we  provide  an  overview  of  the  data  we  used  to  analyze  the  effect 
of  program  and  contract  variances,  program  segment,  contract  type,  and  other  variables 
on  program  and  contract  cost  during  the  systems  acquisition  phase  of  MDAPs.  In  this 
analysis,  we  seek  to  close  the  existing  gaps  in  the  understanding  of  program  and  contract 
management  data.  Cross-referencing  contract  variances  with  program  variances  provides 
acquisition  professionals  with  a  more  complete  picture  of  program  and  contract  changes 
that  occur  during  both  the  EMD  and  production  phases  of  the  acquisition  process. 

We  first  discuss  the  cross-sectional,  time-series  data  we  collected  from  available 
resources.  The  collection  of  this  data  permitted  us  to  construct  a  database  containing  the 
numerical  and  categorical  data  that  we  used  in  the  descriptive  statistical  analysis  in  this 
chapter.  We  provide  the  descriptive  statistics  to  explore  the  initial  data  and  to  identify 
initial  patterns  among  programs  and  contracts.  These  descriptive  statistics  enhance 
readers’  understanding  of  the  data  we  employed  in  the  formal  empirical  analysis 
contained  in  Chapter  V,  in  which  exploratory  empirical  models  are  estimated. 

We  then  provide  a  brief  overview  of  cross-sectional,  time-series  analysis  for 
reader  understanding  of  the  models  that  we  estimated.  Finally,  we  present  a  simple  flow 
diagram  depicting  the  relationships.  Our  objective  was  to  determine  whether  the  data 
were  consistent  so  that,  in  turn,  we  could  determine  the  effects  of  contract  types  and 
variances  on  program  variances,  and  if  the  data  were  not  consistent,  based  on  statistical 
analysis,  we  developed  an  alternative  exploratory  model.  We  present  this  alternative 
structure  and  the  final  conclusions  of  our  analysis  in  Chapter  VI. 


B.  DATA  COLLECTION 


In  this  research,  we  used  quantitative  analysis  to  answer  the  research  questions. 
Quantitative  studies  require  that  a  number  of  assumptions  be  made.  These  include  the 
assumptions  that  the  process  used  is  statistically  reliable  and  operationally  meaningful 
and  that  legitimate  generalizations  can  be  made  from  the  sample  to  predict,  explain,  and 
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understand  the  population  (Creswell,  1994).  The  method  we  used  was  to  collect  and 
model  our  data  in  the  most  unbiased  manner  possible.  We  also  examined  normality  and 
other  characteristics  of  the  data.  In  our  research,  the  population  was  all  MDAPs  and  the 
sample  was  the  dataset  available  that  fit  the  required  analysis  techniques  we  applied  in 
order  to  objectively  make  observations  and  generalizations  about  MDAPs. 

Our  analysis  in  this  study  relied  on  data  contained  in  DAMIRS  and  FPDS. 
Because  each  of  the  Services  use  these  databases  to  document  acquisition  information, 
the  extensive  number  of  SARs  provided  the  most  meaningful  data.  We  downloaded  each 
program  SAR  from  DAMIRS  and  transferred  the  SAR  into  Excel,  breaking  the 
information  in  the  SARs  down  by  program  and  contract  to  allow  us  to  analyze  the 
program  and  contract  variables.  When  multiple  contract  line  item  numbers  (CLINs)  that 
are  reportable  in  a  SAR  were  present  on  a  contract,  we  recorded  each  one  in  the  database 
as  a  separate  contract  observation,  in  keeping  with  the  practice  used  for  SARs.  We 
conducted  further  data  collection  in  FPDS  to  determine  the  predominant  contract  type 
and  obtain  data  missing  in  the  SARs.  Figure  8  shows  the  number  and  percentage  of  the 
different  contract  types. 
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Figure  8.  Actual  Contract  Type  Breakdown 

We  grouped  contract  type  into  six  categories:  FFP,  FPIF,  CPIF,  CPAF,  CPFF,  and 
Hybrid/Other.  The  Hybrid/Other  category  only  contained  5%  of  the  contracts  and 
included  FPIS,  FPEPA,  Time  and  Materials,  hybrid  contract  types  with  a  50-50  split  in 
contract  type,  and  any  indeterminate  predominant  contract  type.  A  50-50  split  in  contract 
type  was  found  in  the  database  for  one  contract;  since  this  contract  was  shown  as  a 
perfect  split  between  CPAF  and  CPIF  in  the  SARs,  determining  a  predominant  contract 
type  was  not  possible.  Several  indeterminate  predominant  contract  types  were  also 
observed;  these  were  most  common  when  a  contract  had  been  awarded,  but  not  yet 
definitized.  Some  undefinitized  contracts  in  the  data  could  be  the  result  of  the  program 
office  and  contractor  still  negotiating  predominant  contract  type,  particularly  in  the  case 
of  letter  contracts. 
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We  also  broadly  grouped  contract  type  into  cost-plus  and  fixed-price  to  compare 
these  two  categories.  The  exact  number  of  each  contract  type  for  each  program  can  be 
seen  in  Table  2,  along  with  the  total  number  of  contracts  for  each  program.  Table  2  also 
includes  the  predominant  contract  type  in  EMD,  RDT&E,  and  production.  We 
determined  predominant  contract  type  based  on  the  total  current  contract  price  for  each 
program.  We  summed  the  total,  current  contract  price  for  each  category  for  the  EMD  and 
the  production  phases,  and  for  RDT&E  appropriation  contracts.  The  contract  type 
associated  with  the  largest  total,  current  contract  price  was  designated  as  the  predominant 
contract  type.  The  EMD  predominant  contract  type  variable  lacked  variability,  since  no 
program  had  a  predominant  fixed-price  orientation  during  EMD.  Due  to  a  lack  of 
variability  in  predominant  contract  type  during  the  EMD  phase,  we  extracted  the 
predominant  contract  type  associated  with  RDT&E  appropriations  and  treated  RDT&E  as 
a  proxy  for  EMD,  recognizing  that  using  a  proxy  limited  our  ability  to  generalize  and 
draw  substantial  conclusions  from  our  results.  Using  a  proxy  for  predominant  EMD 
contract  type  was  thought  to  be  better  than  not  attempting  to  model  predominant  EMD 
contract  type  at  all. 
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Table  2.  MDAP  Number  of  Contracts,  Predominant  Contract  Type  During 


EMD  and  Production,  and  RDT&E  Appropriation 


Contract  Type 

Tot 

al 

Predominant  Contract 
Type 

Program 

CP 

AF 

CP 

FF 

CPI 

F 

FF 

P 

FPI 

F 

Othe 

r 

EMD 

RDT& 

E 

Prod 

AB3A 

0 

0 

12 

0 

0 

0 

12 

CPIF 

CPIF 

CPIF 

AEHF 

17 

0 

0 

0 

0 

0 

17 

CPAF 

CPAF 

CPA 

F 

AGM-88E  AARGM 

0 

0 

7 

2 

2 

0 

11 

CPIF 

CPIF 

CPIF 

AIM-9X 

0 

5 

6 

18 

0 

0 

29 

CPIF 

CPIF 

FFP 

B-2  RMP 

0 

0 

2 

0 

2 

4 

8 

CPIF 

Other 

FPIF 

C-130  AMP 

7 

0 

0 

2 

0 

0 

9 

CPAF 

CPAF 

FFP 

C-5  RERP 

3 

0 

0 

6 

0 

14 

23 

CPAF 

CPAF 

Othe 

r 

CEC 

27 

6 

0 

36 

1 

0 

70 

CPAF 

CPAF 

CPA 

F 

CH-47F 

0 

9 

7 

15 

7 

0 

38 

CPIF 

FFP 

FFP 

E-2D  AHE 

7 

0 

3 

3 

4 

0 

17 

CPAF 

CPAF 

CPA 

F 

EA-18G 

7 

4 

3 

2 

3 

12 

31 

Other 

CPAF 

FPIF 

EXCALIBUR 

0 

0 

6 

8 

0 

0 

14 

Other 

CPIF 

CPIF 

F-22 

13 

9 

1 

47 

0 

9 

79 

CPAF 

CPAF 

FFP 

FBCB2 

8 

0 

3 

8 

5 

0 

24 

None 

CPAF 

FFP 

H-1  UPGRADES 

11 

2 

0 

20 

0 

0 

33 

CPAF 

CPAF 

FFP 

HIMARS 

0 

0 

0 

25 

0 

0 

25 

None 

None 

FFP 

JASSM 

12 

5 

6 

23 

2 

0 

48 

CPFF 

CPAF 

FFP 

JDAM 

4 

0 

0 

19 

0 

0 

23 

None 

CPAF 

FFP 

JSOW  BASELINE 

0 

0 

2 

5 

3 

0 

10 

None 

CPIF 

FFP 

LUH 

0 

0 

0 

5 

0 

0 

5 

None 

None 

FFP 

MH-60R 

0 

9 

15 

27 

2 

4 

57 

CPFF 

CPIF 

FFP 

MH-60S 

0 

0 

18 

29 

0 

0 

47 

None 

CPIF 

FFP 

MUOS 

1 

0 

6 

0 

3 

1 

11 

CPIF 

CPIF 

CPIF 

NAS 

0 

0 

0 

13 

0 

0 

13 

None 

FFP 

FFP 

P-8A 

7 

0 

0 

2 

0 

0 

9 

CPAF 

CPAF 

CPA 

F 

PATRIOT  PAC-3 

0 

0 

28 

23 

0 

4 

55 

CPIF 

CPIF 

FFP 

SM-6 

7 

0 

0 

0 

1 

0 

8 

CPAF 

CPAF 

CPA 

F 

SSN  774 

2 

25 

5 

0 

29 

0 

61 

None 

FPIF 

FPIF 

1.  TACTICAL 

0 

5 

0 

15 

6 

0 

26 

CPFF 

CPFF 

FFP 
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TOMAHAWK 


UH-60M 
V- 22 
Total 


8 

13 

154 


85 


137 


6 

38 

397 


0 

13 

83 


0 

0 

48 


18  CPAF 
73  None 
904 


CPAF 

CPAF 


FFP 

FPIF 


The  major  program  information  we  extracted  from  the  SARs  included  the  DoD 
component,  the  milestones,  the  projected  program,  and  the  variances.  The  contract 
information  included  initial,  current,  and  final  price  data,  and  earned  value  variances. 
From  this  collection  process,  we  built  an  extensive  dataset.  We  developed  additional 
variables  to  conduct  further  analysis.  We  categorized  the  programs  into  four  segments: 
(1)  Aircraft,  including  planes  and  helicopters;  (2)  Missile,  weapons  and  ammunition;  (3) 
C4ISR,  including  radar,  satellite,  and  communication  systems;  and  (4)  Ship  and 
submarine.  Figure  9  shows  the  breakdown  with  the  number  of  programs  in  each 
segment. 


12 


Aircraft  (plane,  Missile,  weapons,  C4ISR  Ship,  submarine 
helicopter)  ammunition 

MDAP  Segment 


Figure  9.  Number  of  MDAPs  by  Segment 

The  database  consists  of  cross-sectional,  time-series  data.  The  cross  section 
includes  multiple  programs  and  their  attendant  contracts  during  a  particular  year.  The 
time  series  consists  of  individual  programs  and  their  attendant  contracts  identified  by 
SAR  data  over  a  several-year  period.  A  current  list  of  active  MDAPs  consisted  of  84 
programs.  In  order  to  perform  a  thorough  analysis  of  the  preproduction  variance  effect 
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on  production,  we  chose  only  those  MDAPs  that  included  pre-  and  post-MS  B  data.  To 
observe  the  variance  effects  over  time,  a  minimum  of  a  five-year  span  in  data  availability 
was  necessary.  This  narrowed  the  number  of  available  programs  to  31,  listed  in  Table  3 
with  their  full  popular  name,  branch  of  service,  segment  category,  and  type.  The  segment 
codes  are:  Aircraft  =  A;  Missile,  weapons,  and  ammunition  =  M;  C4ISR  =  R;  and  Ship 
and  submarine  =  S.  The  program  type  is  either  modification  (Mod)  or  new  start  (New). 
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Table  3.  Programs  Included  in  the  Analysis 


Program 

Full  Name 

Servic 

e 

Segm 

ent 

Ty 

pe 

AB3A 

Longbow  Apache — Block  IIIA 
Remanufacture 

Army 

A 

Mo 

d 

C-130  AMP 

C-130  Avionics  Modernization 

Program 

Air 

Force 

A 

Mo 

d 

C-5  RERP 

C-5  Reliability  Enhancement  and 
Reengining  Program 

Air 

Force 

A 

Mo 

d 

CH-47F 

CH-47F  Improved  Cargo  Helicopter 

Army 

A 

Mo 

d 

E-2D  AHE 

E-2  Advanced  Hawkeye 

Navy 

A 

Mo 

d 

EA-18G 

EA-18G  Growler 

Navy 

A 

Ne 

w 

F-22 

F-22  Advanced  Tactical  Fighter 

Air 

Force 

A 

Ne 

w 

H-1  UPGRADES 

H-1  UPGRADES  (4BW/4BN) 

Navy 

A 

Mo 

d 

LUH 

Light  Utility  Helicopter 

Army 

A 

Ne 

w 

MH-60R 

MH-60R  Multi-Mission  Helicopter 

Navy 

A 

Ne 

w 

MH-60S 

MH-60S  Fleet  Combat  Support 
Helicopter 

Navy 

A 

Mo 

d 

P-8A 

P-8A  POSEIDON 

Navy 

A 

Ne 

w 

UH-60M 

UH-60M  BLACK  HAWK 

Army 

A 

Mo 

d 

V- 22 

V-22  Joint  Services  Advanced 

Vertical  Lift  Aircraft — Osprey 

Navy 

A 

Ne 

w 

AGM-88E  AARGM 

AGM-88E  Advanced  Anti-Radiation 
Guide  Missile 

Navy 

M 

Mo 

d 

AIM-9X 

AIM-9X  Air-to-Air  Missile 

Navy 

M 

Ne 

w 

EXCALIBUR 

Excalibur  Precision  155mm 

Projectiles 

Army 

M 

Ne 

w 

HI  MARS 

High  Mobility  Artillery  Rocket  System 

Army 

M 

Ne 

w 

JASSM 

Joint  Air-to-Surface  Standoff  Missile 

Air 

Force 

M 

Ne 

w 

JDAM 

Joint  Direct  Attack  Munition 

Air 

Force 

M 

Ne 

w 

JSOW  BASELINE 

Joint  Standoff  Weapon  Baseline 

Variant  and  Unitary  Warhead  Variant 

Navy 

M 

Ne 

w 

PATRIOT  PAC-3 

Patriot  Advanced  Capability-3 

Army 

M 

Ne 

w 

SM-6 

Standard  Missile-6 

Navy 

M 

Ne 

w 

2.  TACTICAL 
TOMAHAWK 

Tactical  Tomahawk  R/UGM-109E 

Navy 

M 

Ne 

w 

AEHF 

Advanced  Extremely  High  Frequency 
Satellite 

Air 

Force 

R 

Ne 

w 

B-2  RMP 

B-2  Radar  Modernization  Program 

Air 

Force 

R 

Mo 

d 

CEC 

Cooperative  Engagement  Capability 

Navy 

R 

Ne 
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w 

FBCB2 

Force  XXI  Battle  Command  Brigade 
and  Below  Program 

Army 

R 

Ne 

w 

MUOS 

Mobile  User  Objective  System 

Navy 

R 

Ne 

w 

NAS 

National  Airspace  System 

Air 

Force 

R 

Ne 

w 

SSN  774 

SSN  774  Virginia  Class  Submarine 

Navy 

S 

Ne 

w 

We  performed  normality  tests  on  the  variables  in  the  dataset.  There  are  a  number 


of  graphical  and  non-graphical  normality  tests.  A  simple  graphical  test  of  normality  is  a 
histogram.  We  graphed  a  histogram  for  each  dependent  variable  to  determine  whether 
the  variance  measures  were  normally  distributed.  The  variance  measures  that  were  the 
primary  focus  of  the  research  each  displayed  a  bell  shape  similar  to  a  normal  distribution, 
but  with  excessive  kurtosis  that  caused  the  data  to  be  non-normal.  The  statistical 
methods  used  in  this  research  were  sufficiently  robust  to  accommodate  the  use  of 
variables  exhibiting  non-normality.  Further  explanation  of  the  non-normality  exhibited 
can  be  found  in  Appendix  B. 


C.  DESCRIPTIVE  STATISTICS 


The  questions  we  address  in  this  research  required  quantitative  analysis. 
Typically,  the  first  step  undertaken  in  an  empirical  analysis  is  to  establish  descriptive 
statistics  (Hair,  Black,  &  Anderson,  2009).  In  this  research,  we  examined  two  basic  data 
types:  nominal  data  and  interval  data.  Each  type  of  data  required  a  different  type  of 
analysis. 


Interval  data  are  the  least  restrictive  of  these  two  data  types.  Also  known  as  scale 
data,  interval  data  include  observations  that  can  be  compared  numerically.  Program  cost 
variances,  schedule  variances,  and  engineering  variances,  and  contract  cost  variances  and 
schedule  variances  are  examples  of  interval  data.  The  program  variances  are  reported  in 
constant  2010  millions  of  dollars,  and  have  the  same  scale.  The  contract  variances  are 
reported  in  then-year  millions  of  dollars  over  the  course  of  the  contract.  Conversion  of 
the  contract  variances  to  constant  2010  millions  of  dollars  was  not  attempted,  since 
measuring  the  period  of  performance  and  the  weighted  spend  rate  for  observation  would 
have  been  prohibitively  complicated.  Some  of  the  most  common  measures  used  to 
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examine  interval  data  include  the  mean,  range,  and  standard  deviation.  Statistical 
techniques  valid  for  nominal  data  are  also  frequently  used. 

Nominal  data  include  observations  by  category  that  cannot  be  placed  in  a  logical 
order.  The  type  of  contract  used  for  an  acquisition  is  an  example  of  nominal  data. 
Although  contracts  exhibit  risk  along  a  scale  in  a  defined  order,  not  all  contract  types  are 
practical  for  a  given  acquisition,  so  it  is  not  universally  valid  to  generalize  that  CPFF 
follows  CPAF,  which  follows  CPIF.  This  precludes  the  consideration  of  contracts  as 
ordinal  data.  Nominal  data  are  examined  for  mode  and  frequency  of  occurrence,  and 
typically  are  used  to  construct  categorical  variables  (0-1)  for  each  nominal  category.  For 
nominal  data,  cross-tabulation  (crosstab)  tables  can  be  a  helpful  descriptive  statistic  for 
comparing  the  nominal  categories  of  one  variable  with  the  categories  of  other  variables. 
An  example  of  a  crosstab  table  is  Table  2,  showing  the  number  of  contract  types  for  each 
program. 

In  this  research,  we  examined  data  to  determine  the  causes  of  program  and 
contract  cost  growth.  A  critical  first  step  in  our  statistical  analysis  was  the  application  of 
descriptive  statistics  to  the  data.  We  examined  nominal  and  interval  data  using 
appropriate  statistical  techniques. 

D.  THEORETICAL  FOUNDATION 

Following  the  completion  of  the  first  step  of  applying  descriptive  statistics,  we 
used  several  statistical  techniques  to  further  examine  the  cross-sectional,  time-series 
database  we  developed  from  DAMIRS.  The  primary  technique  we  employed  was 
multiple  regression  analysis.  We  used  multiple  regression  analysis  to  model  the 
association  between  each  included  explanatory  variable  and  the  dependent  variable  of  our 
cross-sectional,  time-series  dataset.  After  conducting  initial  regressions  using  a  structure 
based  on  our  acquisition  experience,  we  revised  the  models,  taking  into  account  both  the 
statistical  properties  of  the  models  and  our  operational  knowledge  of  acquisition. 

The  use  of  a  cross-sectional,  time-series  analysis  allowed  us  to  introduce  multiple 
observations  of  the  acquisition  program  and  the  associated  contracts  into  the  complete 
dataset  across  the  program  reporting  times  identified  in  the  SARs.  In  this  type  of  cross- 
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sectional,  time-series  analysis,  the  statistical  tests  are  identical  to  those  applicable  in 
traditional  regression  analysis.  Based  on  our  experience,  we  felt  that  the  use  of  fixed- 
price  contracts  during  EMD  would  affect  program  outcomes;  however,  there  were  no 
programs  in  which  fixed-price  contracts  were  the  predominant  contract  type  during  EMD. 
Because  there  were  programs  in  which  the  predominant  contract  type  was  fixed  price 
during  the  production  phase  of  the  acquisition  process,  we  examined  this  categorical 
variable.  We  extracted  the  contract  type  data  associated  with  RDT&E  appropriations  in 
order  to  compare  it  to  the  predominant  contract  type  in  the  production  phase. 


E.  CONCEPTUAL  MODELS 


As  indicated,  our  objective  in  this  research  is  to  determine  the  likely  effects  of  the 
increased  usage  of  fixed-price  contracts  during  the  acquisition  process  on  program  and 
contract  costs.  In  order  to  meet  this  objective,  we  designed  an  interrelationship  model  of 
the  variables.  Figure  10  contains  the  basic  variables  and  displays  the  conceptual 
hypotheses  of  how  contract  type  affects  contract  variances  and,  ultimately,  how  it  affects 
program  variances.  These  variances  are  measured  in  dollars  over  time,  and  the  factors 
also  correspond  to  the  independent  and  dependent  variables  we  examined  in  the  cross- 
sectional,  time-series  analysis. 


Contract 

Variances 


Program 

Variances 


Contract 

Type 


Figure  10.  Flow  Diagram  of  Conceptual  Hypothesis 
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F. 


SUMMARY 


In  this  chapter,  we  described  our  data  collection  process  and  displayed  some  key 
features  of  this  data.  We  also  provided  a  brief  discussion  of  the  statistical  methods  we 
used  for  our  analysis.  These  methods  included  descriptive  statistics  and  multiple 
regression  analysis.  In  the  next  chapter,  we  continue  our  discussion  of  methodology  and 
report  the  results  of  the  statistical  analysis  we  conducted. 
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V.  STATISTICAL  METHODOLOGY  AND  RESULTS 


A.  OVERVIEW 

In  this  chapter,  we  initially  provide  a  discussion  of  regression  analysis  as  it  relates 
to  our  exploratory  models.  We  then  provide  an  explanation  of  the  results  from  our 
analyses  and  answer  the  research  questions  we  proposed  in  Chapter  I.  We  identify  the 
relationship  between  both  program  and  contract  cost,  and  program  and  contract 
variances,  as  programs  progress  through  development  to  production.  Understanding 
these  interrelationships  should  help  acquisition  professionals  better  manage  cost, 
schedule,  and  technical  risk  in  MDAPs. 

B.  MULTIPLE  REGRESSION  ANALYSIS 

The  primary  analytical  tool  used  in  this  chapter  is  multiple  regression  analysis. 
This  technique  estimated  the  effect  of  a  number  of  specified  explanatory  variables  on  a 
particular  dependent  variable.  In  this  analysis,  we  employed  historical  observational 
data,  and  researchers  can  expect  these  variables  to  be  correlated  with  each  other. 

However,  multiple  regression  analysis  is  designed  to  deal  with  this  issue.  When 
one  of  the  explanatory  variables  changes,  the  others  are  statistically  held  constant  so  the 
effect  of  the  changing  explanatory  variables  on  the  dependent  variables  is  isolated. 

It  is  important  to  appreciate  that  a  multiple  regression  model  with  a  low  R2  can 
still  have  important  uses.  When  /-statistics  are  statistically  significant  at  the  same  time  as 
the  R  is  low,  the  model  can  be  used  to  accurately  estimate  the  change  in  the  dependent 
variables  when  a  particular  policy  variable  changes.  In  this  situation,  we  can  have  a  valid 
policy  response  model. 

In  forecasting  models,  the  R2  typically  needs  to  be  large.  However,  it  remains 
true  that  in  policy  response  models  with  low  R2  and  high  /-statistics,  researchers  can 
accurately  predict  the  value  of  dependent  variables  for  a  group  of  cases  with  like  values 
for  the  explanatory  variables,  even  though  there  may  be  significant  uncertainty  associated 


ACQUISITION  RESEARCH  PROGRAM 

GRADUATE  SCHOOL  OF  BUSINESS  &  PUBLIC  POLICY 

NAVAL  POSTGRADUATE  SCHOOL 


-61  - 


with  predicting  for  an  individual  case.  It  is  frequently  the  case  that  policy  is  made  based 
on  what  is  likely  to  happen  on  average  (Gilster,  1970). 

In  this  analysis,  we  also  make  extensive  use  of  categorical  variables  that  are 
formed  from  some  group  and  that  take  on  a  value  of  1  if  some  condition  holds — for 
example,  a  specific  type  of  contract — and  0  otherwise.  In  the  case  of  contract  type,  as  an 
example,  one  of  the  contract  types  is  selected  as  the  reference  category  and  excluded 
from  the  model.  The  size  and  significance  of  the  group’s  included  explanatory  variables 
are  always  evaluated  relative  to  the  excluded  categorical  variables.  Therefore,  if  an 
included  categorical  variable  is  not  statistically  significant,  this  means  that  it  is  not 
significantly  different  from  the  excluded  variable. 

It  is  our  expectation  that  the  current  analysis  will  be  followed  by  subsequent 
analyses,  and  that,  as  a  result,  a  consensus  may  emerge  as  to  the  appropriate  specification 
that  accurately  reflects  the  underlying  causal  structure.  Then,  it  will  be  possible  to 
determine  whether  the  data  are  consistent  with  this  causal  structure.  However,  we  view 
our  regression  analyses  as  exploratory  in  nature,  and  this,  in  turn,  resulted  in  several 
decisions  as  to  how  to  display  the  results. 

For  example,  when  the  binary  variables  that  are  members  of  a  particular  group  are 
all  non-significant,  we  frequently  display  the  results  obtained.  While  an  alternative 
would  be  to  simply  state  that  none  of  the  binary  variables  are  significantly  different  from 
each  other,  including  all  of  the  non- significant  binary  variables  may  aid  future  analysts 
who  build  upon  our  work. 

Also,  a  variable  believed  to  be  important  to  the  analysis  that  has  the  predicted 
sign,  but  which  is  statistically  very  insignificant,  may  be  retained  in  the  model.  This  can 
also  aid  future  analysts  who  believe  this  variable  to  be  important  and  choose  to  include  it 
in  a  model  with  a  different  specification. 


We  have  also  limited  our  exploratory  examination  to  the  estimation  of  a  number 
of  single-equation  regression  models.  Linked  regression  model  that  contain  more  than  a 
single  equation  are  not  included.  For  example,  we  did  not  evaluate  an  empirical  path  in 
which,  say,  contract  schedule  variance  and  other  variables  affect,  say,  program 
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engineering  variance,  which,  in  conjunction  with  other  explanatory  variables,  affects  net 
program  variance.  This  type  of  path-analysis  modeling  awaits  additional  research. 

C.  DEFINITION  OF  VARIABLES 

1.  Types  of  Data 

We  now  provide  more  detail  on  the  types  of  data  employed.  As  discussed  in 
Chapter  IV,  the  variables  used  in  this  research  can  be  grouped  into  two  categories: 
numerical  and  nominal.  Numerical  data  can  be  compared  using  regression  and  other 
means,  but  nominal  data  cannot  be  directly  compared.  Instead,  nominal  variables  must 
be  converted  to  a  series  of  binary  variables,  which  are  also  frequently  called  dummy 
variables.  Accordingly,  dummy  variables  used  in  the  research  end  in  “_D.” 

2.  Binary  Variables 

As  indicated  previously,  binary  variables  indicate  the  presence  or  lack  of  presence 
of  a  category  or  condition.  Two  or  more  categories  may  be  compared  using  dummy 
variables.  For  example,  two  types  of  acquisitions  are  considered  in  this  research:  new 
start  acquisitions  and  modification  acquisitions.  Acquisition  type  is  represented  by  the 
variables  “AcqType_Mod_D”  and  “AcqType_New_D.”  When  the  value  of 
AcqType_Mod_D  is  1,  this  indicates  that  the  MDAP  is  a  modification  program;  when  the 
value  is  0,  the  MDAP  is  a  new  start  program.  When  the  value  of  AcqType_New_D  is  1, 
this  indicates  that  the  MDAP  is  a  new  start  program.  Because  the  sum  of  these  two 
variables  necessarily  equations  1,  one  of  the  two  would  be  selected  for  explicit  use  in  the 
regression,  and  the  other  would  be  the  reference  variable. 

As  indicated,  when  examining  the  effect  of  a  dummy  variable,  readers  should 
keep  in  mind  the  variable’s  reference.  The  reference  for  AcqType  Mod  D  is 
AcqType_New_D.  Therefore,  for  regressions  including  AcqTypeModD,  the  effect  of 
a  modification  acquisition  relative  to  the  effect  of  a  new  start  acquisition  is  shown  as  the 
regression  coefficient.  This  coefficient  measures  the  difference  found  in  the  data 
between  when  AcqTypeModD  equals  one  and  when  it  equals  zero. 
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The  remaining  dummy,  categorical,  and  scale  variables  mentioned  in  the  model 
are  listed  below,  broken  down  by  type  and  program  or  contract  relationship. 

a.  Additional  Program  Variables 

EMDD:  1  indicates  the  presence  of  the  EMD  phase;  0  indicates  the 
presence  of  the  production  phase. 

RDTE  D:  1  indicates  the  presence  of  an  RDT&E  appropriation;  0 
indicates  another  appropriation,  typically  procurement. 

PROC  D:  1  indicates  the  presence  of  the  procurement  phase;  0  indicates 
another  appropriation,  typically  RDT&E. 

AcqSegment_Aircraft_D:  1  indicates  the  presence  of  the  Aircraft  (planes 
and  helicopters)  acquisition  segment;  0  indicates  otherwise. 

AcqSegment_C4ISR_D:  1  indicates  the  presence  of  the  C4ISR  acquisition 
segment;  0  indicates  otherwise. 

AcqSegment_Missiles_D:  1  indicates  the  presence  of  the  Missiles 
acquisition  segment;  0  indicates  otherwise. 

AcqSegment_Ships_D:  1  indicates  the  presence  of  the  Ships  acquisition 
segment;  0  indicates  otherwise. 

EMD  FFP  D:  1  indicates  FFP  as  the  predominant  contract  type  during 
EMD;  0  indicates  otherwise. 

EMD  FPIF  D:  1  indicates  FPIF  as  the  predominant  contract  type  during 
EMD;  0  indicates  otherwise. 

EMD  CPIF  D:  1  indicates  CPIF  as  the  predominant  contract  type  during 
EMD;  0  indicates  otherwise. 

EMD  CPAF  D:  1  indicates  CPAF  as  the  predominant  contract  type 
during  EMD;  0  indicates  otherwise. 
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EMDCPFFD:  1  indicates  CPFF  as  the  predominant  contract  type  during 
EMD;  0  indicates  otherwise. 

EMD  CONTOTH  D:  1  indicates  Hybrid/Other  as  the  predominant 
contract  type  during  EMD;  0  indicates  otherwise. 

RDTEFFPD:  1  indicates  FFP  as  the  predominant  contract  type  for 
RDT&E;  0  indicates  otherwise. 

RDTE  FPIF  D:  1  indicates  FPIF  as  the  predominant  contract  type  for 
RDT&E;  0  indicates  otherwise. 

RDTE  CPIF  D:  1  indicates  CPIF  as  the  predominant  contract  type  for 
RDT&E;  0  indicates  otherwise. 

RDTE  CPAF  D:  1  indicates  CPAF  as  the  predominant  contract  type  for 
RDT&E;  0  indicates  otherwise. 

RDTE  CPFF  D:  1  indicates  CPFF  as  the  predominant  contract  type  for 
RDT&E;  0  indicates  otherwise. 

RDTE  CONTOTH  D:  1  indicates  Hybrid/Other  as  the  predominant 
contract  type  for  RDT&E;  0  indicates  otherwise. 

PROD  FFP  D:  1  indicates  FFP  as  the  predominant  contract  type  during 
production;  0  indicates  otherwise. 

PROD  FPIF  D:  1  indicates  FPIF  as  the  predominant  contract  type  during 
production;  0  indicates  otherwise. 

PROD  CPIF  D:  1  indicates  CPIF  as  the  predominant  contract  type  during 
production;  0  indicates  otherwise. 

PROD  CPAF  D:  1  indicates  CPAF  as  the  predominant  contract  type 
during  production;  0  indicates  otherwise. 

PROD  CPFF  D:  1  indicates  CPFF  as  the  predominant  contract  type 
during  production;  0  indicates  otherwise. 
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PROD  CONTOTH  D:  1  indicates  Hybrid/Other  as  the  predominant 
contract  type  during  production;  0  indicates  otherwise. 

EMDBasicContractTypeCPD:  1  indicates  a  cost-plus  basic  contract 
type  as  the  predominant  basic  contract  type  during  EMD;  0  indicates  otherwise. 

EMD  BasicContractType  FP  D:  1  indicates  a  fixed-price  basic  contract 
type  as  the  predominant  basic  contract  type  during  EMD;  0  indicates  otherwise. 

RDTE  BasicContractType  CP  D:  1  indicates  a  cost-plus  basic  contract 
type  as  the  predominant  basic  contract  type  for  RDT&E;  0  indicates  otherwise. 

RDTE  BasicContractType  FP  D:  1  indicates  a  fixed-price  basic  contract 
type  as  the  predominant  basic  contract  type  for  RDT&E;  0  indicates  otherwise. 

PROD  BasicContractType  CP  D:  1  indicates  a  cost-plus  basic  contract 
type  as  the  predominant  basic  contract  type  during  production;  0  indicates  otherwise. 

PROD  BasicContractType  FP  D:  1  indicates  a  fixed-price  basic  contract 
type  as  the  predominant  basic  contract  type  during  production;  0  indicates  otherwise. 


b.  Contract  Variables 

FFPD:  indicates  a  contract  is  FFP;  0  indicates  otherwise. 

FPIF_D:  indicates  a  contract  is  FPIF;  0  indicates  otherwise. 

CPIF  D:  indicates  a  contract  is  CPIF;  0  indicates  otherwise. 

CPAF_D:  indicates  a  contract  is  CPAF;  0  indicates  otherwise. 

CPFF  D:  indicates  a  contract  is  CPFF;  0  indicates  otherwise. 

CONTOTH  D:  indicates  a  contract  is  Hybrid/Other;  0  indicates 


otherwise. 

FFPEMDCaseContractTypeD:  1  indicates  an  FFP  contract  during 
EMD;  0  indicates  otherwise.  As  such,  the  coefficients  of  FFP  D  and 
FFP  EMD  CaseContractType  D  can  be  summed  to  determine  the  performance  of  FFP 
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during  EMD.  The  remaining  Case  Contract  Type  variables  should  be  interpreted  in  the 
same  manner. 

FPIFEMDCaseContractTypeD:  1  indicates  an  FPIF  contract  during 
EMD;  0  indicates  otherwise. 

CPIFEMDCaseContractTypeD:  1  indicates  a  CPIF  contract  during 
EMD;  0  indicates  otherwise. 

CPAF  EMD  CaseContractType  D:  1  indicates  a  CPAF  contract  during 
EMD;  0  indicates  otherwise. 

CPFF  EMD  CaseContractType  D:  1  indicates  a  CPFF  contract  during 
EMD;  0  indicates  otherwise. 

CONTOTH  EMD  CaseContractType  D:  1  indicates  a  Hybrid/Other 
contract  during  EMD;  0  indicates  otherwise. 

3.  Categorical  Variables  Without  Dummy  Equivalents 

a.  Program  Variable 

ProgramID:  A  unique  program  identification  number  associated  with  a 
contract  representing  a  single  scope  of  work. 

b.  Contract  Variable 

ProgramandContractID:  A  unique  contract  identification  number 
associated  with  a  contract  representing  a  single  scope  of  work.  This  identification 
number  is  separate  and  distinct  from  a  contract’s  contract  number. 

4.  Scale  Variables 

a.  Program  Variables 

MilestoneBscheduleachieved_Fractional_Year:  The  year  in  which 
Milestone  B  was  achieved,  represented  numerically  as  whole  years  with  decimal  partial 
years. 
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ProgramFractionalYear:  The  number  of  years  since  formal  program 
inception  (typically  at  Milestone  B),  represented  numerically  as  whole  years  with  decimal 
partial  years. 

SARBaselineProdEstConstant2010$M:  The  program  baseline  cost 
estimate.  This  variable  and  the  remaining  variables  labeled  Constant2010$M  are 
presented  in  constant  2010  millions  of  dollars. 

Program  Schedule  Variance,  Constant2010$M:  The  current  period  cost 
variance  attributable  to  schedule  changes. 

Program  Engineering  Variance,  Constant2010$M:  The  current  period  cost 
variance  attributable  to  engineering  changes. 

SubtotalCurrentChangesConstant2010$M:  The  current  total  cost  variance. 

CECostVarianceConstant2010$M:  This  is  the  expected  program  cost 
through  the  end  of  production  and  is  the  sum  of  the  program  baseline  cost  estimate,  prior 
cost  variance,  and  current  cost  variance. 

Program  Cost  Variance,  Constant2010$M:  The  current  period  cost 
variance  attributable  to  any  program  change,  with  the  exception  of  schedule  and 
engineering  changes.  This  variance  is  also  presented  in  constant  2010  millions  of  dollars. 

CurrentLessInitialProgramCost:  The  current  estimate  of  program  cost 
minus  the  initial  estimate  of  program  cost,  presented  in  constant  2010  millions  of  dollars. 

b.  Contract  Variables 

TargetInitialContractPrice$M:  The  initial  target  contract  price  target  in 
millions  of  dollars. 

TargetCurrentContractPrice$M:  The  current  target  contract  price  in 
millions  of  dollars. 

ProgramManagerEstimatedPriceAtCompletion$M:  The  current  program 
manager’s  contract  estimated  price  at  completion  in  millions  of  dollars. 
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Contract  Cost  Variance:  The  contract  earned  value  cost  variance  in 


millions  of  dollars. 

Contract  Schedule  Variance:  The  contract  earned  value  schedule  variance 
in  millions  of  dollars. 

CurrentLessInitialContractPrice:  The  current  estimate  of  contract  price 
minus  the  initial  estimate  of  contract  price  in  millions  of  dollars. 

ContractEACLessInitialContractPrice:  The  current  contract  estimate  at 
completion  minus  the  initial  estimate  of  contract  price  in  millions  of  dollars. 

ContractEACLessCurrentContractPrice:  The  current  contract  estimate  at 
completion  minus  the  current  estimate  of  contract  price  in  millions  of  dollars. 

For  an  alphabetized  list  of  the  variables  used  see  Appendix  A. 

D.  RESEARCH  QUESTIONS 
1.  Primary  Questions 

1.  What  effect  does  fixed-price  R&D  have  on  production  cost,  schedule,  and 
technical  performance? 

Table  2  in  Chapter  IV  showed  the  predominant  contract  type  computed  for  RDT&E 
appropriation.  There  were  three  programs  (CH-47F,  NAS,  and  SSN  774)  with 
predominant  fixed-price  contracts  for  RDT&E.  In  order  to  answer  this  research  question, 
we  computed  the  average  CV,  SV,  and  EV  for  all  3 1  MDAPs  in  the  sample  to  contrast 
the  effects  of  predominant  fixed-price  RDT&E  contracts  on  the  three  MDAPs.  The 
averages  are  listed  in  Table  4 
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Table  4.  Average  Program  Variances  for  Entire  Sample  and  Predominant  FP 


in  RDT&E  (2010  S 

>M) 

CV 

SV 

EV 

Average  Program  Variance  all 
MDAPs 

3093.6 

366.5 

651.6 

Average  Program  Variance  of 
Predominant  FP  Contracts  in 

RDT&E 

9306.0 

614.0 

685.3 

Table  4  shows  that  the  average  CV  for  the  entire  sample  of  MDAPs  was  $3,093.6 
million,  while  the  average  CV  for  predominant  fixed-price  RDT&E  contracts  was  $9,306 
million,  or  an  average  CV  increase  of  over  $6  billion.  There  were  only  three  programs 
with  predominant  fixed-price  RDT&E  contracts,  which  do  not  make  these  observations 
compelling  evidence.  We  would  assume  that  fixed-price  contracts  would  assist  in 
controlling  program  costs,  but  the  use  of  fixed-price  contracts  during  RDT&E  is 
associated  with  larger  CV. 

Table  4  also  identifies  the  average  SV  on  predominant  fixed-price  RDT&E 
contracts  to  be  approximately  $250  million  higher  than  the  sample,  and  EV  was  only 
higher  by  approximately  $35  million.  These  figures,  although  computed  on  three 
programs,  show  that  the  predominant  use  of  FP  contracts  during  RDT&E  is  associated 
with  higher  SV. 

2.  Do  different  segments  of  MDAPs  (e.g.,  fighters,  tanks,  missiles,  satellites) 
exhibit  differing  cost  and  schedule  growth? 

We  categorized  the  31  MDAPs  into  four  segments:  A,  M,  R,  and  S.  These 
segments  are  defined  in  Table  5,  with  the  number  of  programs  for  each  category  shown  in 
the  total  column. 
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Table  5.  MDAP  Segment 

CODE 

SEGMENT  CATEGORIES 

Total 

A 

Aircraft  (plane,  helicopter) 

14 

M 

Missile,  weapons,  ammunition 

10 

R 

C4ISR 

6 

S 

Ship,  submarine 

1 

31 

We  also  categorized  the  MDAPs  into  two  types:  modernization  and  new  start. 
There  were  10  modernization  and  21  new  start  programs.  To  calculate  average  cost 
growth  for  each  segment,  we  calculated  the  total  cost  growth  (including  both  cost 
overruns  and  cost  underruns)  by  subtracting  the  original  SAR  baseline  from  the  latest 
program  estimate,  dividing  by  the  number  of  years,  and  then  dividing  by  the  original 
SAR  baseline.  This  formula  is  shown  in  Figure  1 1 . 

(Latest  Program  Estimate  -  SAR  Original  Baseline) 

Number  of  SAR  Years 

Average  Cost  = _ 

Growth  % 

SAR  Original  Baseline 


Figure  11.  Average  Cost  Growth  Formula 

To  calculate  schedule  growth  over  the  course  of  a  program  for  each  segment,  we 
divided  the  total  schedule  variance  (including  both  schedule  overruns  and  schedule 
underruns)  by  the  current  SAR  baseline.  This  formula  is  shown  in  Figure  12. 

Total  Schedule  Variance 

Average  Schedule  =  _ 

Growth  % 

SAR  Current  Baseline 

Figure  12.  Average  Schedule  Growth  Formula 

We  calculated  each  program’s  cost  and  schedule  growth  and  averaged  this  by 
segment,  shown  in  Table  6  for  easy  comparison.  The  Virginia  Class  Submarine  was  the 


2  The  segment  average  cost  growth  was  computed  from  the  annual  average  growth  percentage  rate  of  each 
program. 
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only  program  in  the  ship  and  submarine  category;  thus,  it  should  not  be  considered  as  an 
average  of  the  segment.  The  other  three  categories  have  a  similar  average  cost  growth, 
with  aircraft  topping  the  segments  with  10.3%  growth.  The  average  schedule  growth 
between  the  three  categories  varied  greatly,  with  aircraft  having  the  least  amount  of 
growth  and  the  C4ISR  segment  averaging  13.4%  growth  in  schedule. 


Table  6.  Average  Program  Cost  and  Schedule  Growth  b; 

y  MDAP  Segment 

#  of 
MDAPs 

SEGMENT  CATEGORIES 

Average  Cost 
Growth 

Average  Schedule 
Growth 

14 

Aircraft  (plane,  helicopter) 

10.33% 

1.57% 

10 

Missile,  weapons,  ammunition 

7.20% 

3.18% 

6 

C4ISR 

9.87% 

13.42% 

1 

Ship,  submarine 

2.96% 

2.65% 

3.  Does  early  cost,  schedule,  or  engineering  variance  serve  as  a  leading  indicator 
of  later-period  CV,  SV,  and/or  EV  in  either  EMD  or  post  Milestone  C? 

Examining  this  research  question  required  the  use  of  multiple  regressions,  which 
we  discuss  in  further  detail  in  Section  D.  The  regressions  applicable  to  the 
interrelationship  of  variances  can  be  found  at  the  end  of  Section  D. 

2.  Secondary  Questions 

1.  What  portion  of  MDAPs  have  fixed-price  incentive  R&D  contracts? 

Based  on  the  3 1  programs  and  904  contracts  in  the  dataset,  there  were  32  fixed- 
price  contracts  issued  during  EMD  from  a  total  of  369  RDT&E-appropriated  CLINs. 
Less  than  9%  of  the  369  RDT&E  contracts  were  fixed-price.  The  initiative  promoting 
fixed-price  R&D  contracts  is  rather  new,  and  the  small  number  of  FP  contracts  is  not 
surprising.  Future  research  is  recommended  to  investigate  the  increase  of  FP  contracts 
used  for  RDT&E  appropriation. 

2.  Is  there  qualitative  information  to  support  the  assertion  that  fixed-price 
contracts  during  the  EMD  phase  hinder  the  identification  of  program  problems? 

The  discussion  of  contract  types  and  preferred  contract  types  by  acquisition  phase 
in  Chapter  II  and  the  discussion  of  the  importance  of  R&D  outcomes  in  Chapter  III  are 
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consistent  with  the  assertion  that  fixed-price  contracts  may  be  associated  with  problem 
identification  in  the  development  process. 

3.  If  the  effect  of  fixed-price  R&D  is  measurable,  are  the  variances  larger  with 
regard  to  cost,  schedule,  or  engineering  during  EMD  and  production? 

This  question  can  be  addressed  as  a  continuation  of  primary  research  question  1 . 
The  CY,  SV,  and  EV  for  predominant  fixed-price  contracts  under  RDT&E  were 
compared  to  the  entire  dataset.  The  average  CV,  SV,  and  EV  for  predominant  FP 
contracts  during  production  was  calculated  and  compared  to  the  dataset  listed  in  Table  7. 

Table  7.  Average  Program  Variances  for  Entire  Sample  and  Predominant  FP 


in  RDT&E  and  Production  (in  $  millions) 


CV 

SV 

EV 

Average  Variance  all  MDAPs 

3093.6 

366.5 

651.6 

Average  Variance  of  Predominant  FP 
Contracts  in  RDT&E 

9306.0 

614.0 

685.3 

Average  Variance  of  Predominant  FP 
Contracts  in  Production 

3889.9 

333.7 

842.9 

The  average  variance  for  predominant  fixed-price  contracts  in  production  was 
very  similar  to  the  entire  dataset  except  for  engineering  variance.  The  average  EV 
increase  of  $191  million  occurring  on  predominant  fixed-price  contracts  during 
production  phase  could  be  attributable  to  increased  technical  risk  experienced  during  the 
production  phase.  Even  fixed-price  contracts  might  not  be  able  to  hedge  the  technical 
risk.  The  ceiling  price  might  provide  incentives  for  a  contract  change  to  be  approved. 
Engineering  changes  occur  due  to  new  technology  upgrades,  redesign,  and  configuration 
changes.  Minimizing  changes  would  be  the  best  prevention  of  poor  engineering 
variances.  Further  analysis  might  relate  this  finding  to  the  associated  contract  type  used 
during  the  EMD  phase. 

4.  Based  on  the  results  found  in  this  research,  can  any  definitive  policy 
recommendations  be  made? 

Our  discussion  in  Chapter  VI  of  the  results  and  recommendations  answers  this 
research  question. 
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E.  MULTIPLE  REGRESSION  ANALYSIS 


We  conducted  a  regression  analysis  to  identify  the  relationship  between  cost  and 
price  changes  at  the  program  and  contract  levels.  These  regressions  partially  formed  the 
basis  of  our  responses  earlier  in  this  chapter  to  the  research  questions  we  posed  in 
Chapter  I.  In  this  section,  we  also  explore  regression  relationships  in  greater  detail  by 
focusing  on  establishing  the  statistical  validity  of  the  data,  given  our  knowledge  of  the 
acquisition  process,  rather  than  on  responding  directly  to  any  specific  research  question. 
This  was  done  using  the  stepwise  method  favored  by  most  statisticians  for  exploratory 
regressions  (Hair  et  al.,  2009).  Following  the  stepwise  method,  we  began  with  initial 
regressions  and  iteratively  improved  these  regressions  until  arriving  at  satisfactory  final 
regressions.  Differing  initial  and  final  regressions  were  run  based  upon  the  relationships 
we  were  attempting  to  quantify. 

We  focused  our  modeling  efforts,  first,  on  relating  cost  and  price  changes  over 
time  at  the  program  and  contract  levels.  Then,  we  related  cost,  schedule,  and  engineering 
variances  at  the  program  level  and  earned  value  cost  and  schedule  variances  at  the 
contract  level.  The  remaining  variables  were  included  because,  based  on  our  knowledge 
of  the  acquisition  process,  we  believed  that  they  could  have  an  effect  on  the  dependent 
variables.  After  developing  program  and  contract  models,  we  examined  models  in 
which  program  data  was  related  to  contract  data. 

1.  Regressions  of  Cost  and  Price  Changes  at  the  Program  and  Contract 
Levels 

We  focused  our  modeling  effort  on  relating  cost  and  price  changes  over  time  at 
the  program  and  contract  levels;  this  focus  arose  from  the  general  trend  in  contract  and 
program  cost  growth  over  time,  as  illustrated  in  Figure  13.  Program  cost  estimates  for  a 
given  SAR  estimate  the  cost  of  an  MDAP  through  the  end  of  production.  Contract  cost 
estimates  for  a  SAR  estimate  the  cost  of  the  contract  through  the  end  of  the  contract  in 
two  forms:  the  current  target  price  and  estimate  at  completion  (EAC).  The  current  target 
price  is  the  current  negotiated  contract  price.  The  EAC  is  an  independent  estimation  of 
the  final  contract  price,  based  upon  the  expected  price  of  definitized  work,  undefinitized 
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work,  and  contract  overruns.  Our  research  uses  the  government  program  manager’s 
EAC. 


Program  Cost  Behavior 

Current 
estimate  at 
most  recent 
SAR 


Contract  Price  Behavior 


EAC 


Figure  13.  Typical  Program  and  Contract  Cost  Growth  Over  Time 

This  typical  pattern  suggests  multiple  avenues  for  studying  the  causes  of  program 
cost  growth,  added  contract  scope,  contract  overruns,  and  their  interrelationships.  For 
this  research,  program  cost  growth  is  defined  as  the  current  estimate  at  time,  t,  minus  the 
initial  cost.  To  better  explore  the  relationship  between  program  cost  growth  and  contract 
price  growth,  we  first  examined  the  relationship  between  program  cost  growth  and 
contract  growth  using  the  same  explanatory  variables  included  in  the  program  cost 
regression,  but  adding  contract-level  variables.  Acquisition  segment,  acquisition  type, 
predominant  basic  contract  type  during  production,  predominant  basic  contract  type  for 
RDT&E,  the  years  since  program  inception,  the  year  that  a  program  achieved  MS  B,  and 
acquisition  phase  were  each  included  as  variables  that  explain  program  cost  growth.  The 
initial  program  cost  growth  model  examined  is  shown  in  Table  8. 
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Table  8.  Initial  Regression  of  Program  Cost  Growth  (Current  Estimate — 


Initial  < 

Cost) 

Model 

Coefficient 

Std.  Error 

f-statistic 

1 

(Constant) 

-35576.462 

123146.728 

-.289 

AcqSegment_Aircraft_D 

-16700.178 

1083.424 

-15.414 

Acq  Seg  m  e  nt_C4 1 S  R_D 

-16792.475 

1048.513 

-16.016 

AcqSegment_Missiles_D 

-16228.618 

1071.70 7 

-15.143 

AcqType_Mod_D 

1741.443 

600.233 

2.901 

PROD_BasicContractType_FP_D 

-587.125 

654.329 

-.897 

RDTE_BasicContractType_FP_D 

-236.511 

800.872 

-.295 

Program_Fractional_Year 

215.176 

55.684 

3.864 

MilestoneBscheduleachieved_Fractional_Year 

25.898 

61.468 

.421 

EMD_D 

-1688.308 

647.847 

-2.606 

Dependent  Variable:  CurrentLessInitialProgramCost 

R2  =  .448 

N  =  827 

After  exploring  multiple  possible  alterations  to  the  model,  we  eliminated  variables 
with  non-significant  /-statistics,  namely  the  predominant  basic  contract  type  for  RDT&E 
and  the  year  that  a  program  achieved  MS  B.  The  resultant  final  regression,  shown  in 
Table  9,  has  significant  /-statistics  and  a  slightly  improved  R2. 

Table  9.  Final  Regression  of  Program  Cost  Growth  (Current  Estimate — Initial 


Cost) 


Model 

Coefficient 

Std.  Error 

^-statistic 

1 

(Constant) 

15820.918 

887.255 

17.831 

AcqSegment Aircraft D 

-16458.915 

700.657 

-23.491 

AcqSegment C4ISR D 

-16371.119 

810.449 

-20.2 

AcqSegment Missiles D 

-16499.36 

691.423 

-23.863 

AcqType Mod D 

1770.771 

482.723 

3.668 

PROD BasicContractType FP D 

-256.852 

514.441 

-0.499 

Program Fractional Year 

186.603 

41.862 

4.458 

EMDD 

-1493.173 

598.052 

-2.497 

Dependent  Variable:  CurrentLessInitialProgramCost 

R2  =  .449 

N  =  879 

The  dummy  variables  in  the  acquisition  segment  category  should  be  viewed  in 
light  of  their  relationship  to  their  reference  dummy  variable,  the  acquisition  segment 
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Ships  dummy  variable.  The  modification  acquisition  type  dummy  should  be  viewed  in 
light  of  its  relationship  to  the  new  start  acquisition  type  dummy.  Therefore,  the 
coefficients  shown  for  each  dummy  should  be  viewed  in  light  of  their  reference — not  in 
absolute  terms.  From  this  regression,  we  can  determine  that  the  acquisition  segment 
dummies  are  each  significantly  different  from  their  reference,  Ships;  each  segment  has 
program  cost  growth  that  is  significantly  less  than  Ships  in  relative  terms.  We  can  also 
determine  that  modification  acquisitions  experience  relatively  higher  cost  growth  than 
new  start  acquisitions.  The  regression  also  shows  that  programs  that  use  a  contract  from 
the  fixed-price  family  of  contracts  (FFP,  FPIF,  FPIS,  or  FPEPA)  during  production 
experience  lower  cost  growth  than  those  that  do  not,  although  not  at  a  significant  level. 
This  variable  is  retained  in  the  model  as  an  aid  to  those  doing  follow-on  work.  The 
regression  also  shows  that  programs  experience  significantly  lower  cost  growth  during 
EMD  than  during  production;  this  is  even  more  significant  given  that  the  regression  also 
shows  that  cost  growth  significantly  increases  as  the  time  since  a  program  has  achieved 
MS  B  increases. 

The  next  model  examined  the  causes  of  contract  price  growth.  We  defined 
contract  price  growth  as  the  contract  estimate  at  completion  (EAC)  minus  the  contract 
initial  target  price.  To  better  explore  the  relationship  between  program  cost  growth  and 
contract  price  growth,  we  examined  the  relationship  between  contract  price  growth  and 
the  variables  we  intended  to  hold  constant  for  our  later  regression  of  program  cost  growth 
on  contract  cost  growth.  Acquisition  phase,  predominant  contract  type  during  EMD, 
acquisition  type,  acquisition  segment,  predominant  basic  contract  type  for  RDT&E,  the 
year  that  a  program  achieved  MS  B,  and  the  years  since  program  inception  were  each 
used  as  variables  for  explaining  contract  price  growth.  The  initial  contract  price-growth 
model  is  shown  in  Table  10. 
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Table  10.  Initial  Regression  of  Contract  Price  Growth  (EAC — Initial  Target) 


Model 

Coefficient 

Std.  Error 

f-statistic 

(Constant) 

4877.025 

17424.792 

.280 

EMD_D 

163.065 

92.847 

1.756 

EMD_CPAF_D 

433.518 

69.075 

6.276 

EMD_CPIF_D 

153.400 

83.308 

1.841 

EMD_CPFF_D 

-229.607 

84.751 

-2.709 

AcqType_Mod_D 

-560.877 

88.288 

-6.353 

AcqSegment_Aircraft_D 

420.955 

115.949 

3.631 

AcqSegment_C4ISR_D 

-115.839 

124.353 

-.932 

AcqSegment_Missiles_D 

18.759 

117.219 

.160 

MilestoneBscheduleachieved_Fractional_Year 

-2.454 

8.707 

-.282 

Program_Fractional_Year 

10.664 

8.308 

1.284 

Dependent  Variable:  ContractEACLessInitialContractPrice 

R2  =  .163 

N  =  891 

After  exploring  multiple  possible  alterations  to  the  model,  we  eliminated  a 
variable  with  a  non-significant  /-statistic,  the  variable  representing  the  year  that  a 
program  achieved  MS  B.  We  also  substituted  specific  contract  type  for  each  contract  for 
predominant  contract  type  during  EMD.  The  resultant  final  regression,  shown  in  Table 
1 1,  has  an  improved  R2. 


Table  11.  Final  Regression  of  Contract  Price  Growth  (EAC — Initial  Target) 


Model 

Coefficient 

Std.  Error 

^-statistic 

(Constant) 

132.035 

168.037 

.786 

AcqSegment Aircraft D 

426.880 

121.139 

3.524 

AcqSegment C4ISR D 

20.141 

127.703 

.158 

AcqSegment Missiles D 

67.671 

121.148 

.559 

AcqType Mod D 

-340.531 

74.202 

-4.589 

Program Fractional Year 

28.110 

6.725 

4.180 

EMD D 

107.048 

91.870 

1.165 

CPIF D 

-346.545 

126.150 

-2.747 

FFP D 

-461.813 

116.671 

-3.958 

FPIF D 

-295.845 

142.762 

-2.072 

CPAF D 

252.499 

124.183 

2.033 

CPFF D 

-402.172 

138.872 

-2.896 

Dependent  Variable:  ContractEACLessInitialContractPrice 

R2  =  . 1 89 

N  =  891 

The  contract  type  dummy  variables  should  be  viewed  in  light  of  their  reference, 
namely,  the  contract  type  dummy  variable  in  the  Hybrid/Other  segment.  The  only 
contract  type  with  more  growth  than  the  reference  contract  type  was  CPAF.  The  reasons 
for  the  relative  price  increase  cannot  be  determined  from  Table  1 1 .  However,  larger  price 
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growth  indicates  that  either  contract  overruns  or  scope  increases  resulted  in  CPAF 
contracts  exhibiting  greater  price  growth.  This  effect  is  still  relative  to  that  displayed  by 
the  reference  contract  type,  Hybrid/Other.  A  similar  interpretation  applies  to  the 
acquisition  segment  variables. 

In  the  next  model,  we  examined  the  causes  of  program  cost  growth,  but  this 
model  also  included  contract  price  growth  as  an  independent  variable.  This  provides  a 
linkage  between  the  program  data  and  the  contract  data.  Again,  we  defined  program  cost 
growth  as  current  cost  minus  initial  cost.  To  better  understand  the  output  of  this 
combined  program  cost  and  contract  price  growth  model,  we  first  ran  the  regressions 
above  to  understand  what  variables  to  include  in  the  combined  regression.  Predominant 
basic  contract  type  during  production,  predominant  basic  contract  type  for  RDT&E, 
acquisition  phase,  contract  type,  acquisition  type,  acquisition  segment,  the  years  since 
program  inception,  and  the  year  that  a  program  achieved  MS  B  were  each  included  in  the 
initial  model.  The  initial  program  cost  model,  including  the  independent  variable  for 
contract  price  growth,  is  shown  in  Table  12. 
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Table  12.  Initial  Regression  of  Program  Cost  Growth  (Current  Estimate — 
Initial  Cost),  Including  the  Independent  Variable  for  Contract  Price 

Growth 


Model 

Coefficient 

Std.  Error 

^-statistic 

(Constant) 

87866.712 

125688.232 

.699 

PROD_BasicContractType_FP_d 

-577.344 

650.660 

-.887 

RDTE_BasicContractType_FP_D 

85.932 

793.588 

.108 

EMD_D 

-1865.279 

651.506 

-2.863 

CPIF_D 

-474.264 

1038.975 

-.456 

FFP_D 

-2526.073 

972.139 

-2.598 

FPIF_D 

-1150.724 

1099.022 

-1.047 

CPAF_D 

-1316.971 

1015.765 

-1.297 

CPFF_D 

-3715.515 

1088.713 

-3.413 

AcqType_Mod_D 

1311.159 

615.329 

2.131 

AcqSegment_C4ISR_D 

-16464.854 

1069.747 

-15.391 

AcqSegment_Missiles_D 

-16054.011 

1084.695 

-14.800 

AcqSegment_Aircraft_D 

-15935.673 

1095.288 

-14.549 

Program_Fractional_Year 

222.874 

58.269 

3.825 

MilestoneBscheduleachieved_Fractional_Year 

-35.031 

62.629 

-.559 

ContractEACLessInitialContractPrice 

-1.350 

.219 

-6.172 

Dependent  Variable:  CurrentLessInitialProgramCost 

R2  =  .491 

N  =  816 

This  initial  regression  of  program  cost  growth,  including  the  independent  variable 
for  contract  price  growth,  had  an  unexpected  negative  coefficient  for  the  variable 
ContractEACLessInitialContractPrice.  A  negative  coefficient  would  indicate  that  as 
contract  price  growth  increases,  program  cost  growth  decreases.  Following  additional 
analysis,  we  obtained  the  result  for  the  final  regression,  shown  in  Table  13. 
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Table  13.  Final  Regression  of  Program  Cost  Growth  (Current  Estimate — Initial 
_ Cost),  Including  the  Independent  Variable  for  Contract  Price  Growth 


Model 

Coefficient 

Std.  Error 

f-statistic 

(Constant) 

17451.266 

2703.592 

6.455 

PROD_BasicContractType_FP_D 

-774.510 

895.329 

-.865 

RDTE_BasicContractType_FP_D 

1627.430 

1481.617 

1.098 

EMDBCpointdate_D 

-2961.289 

885.129 

-3.346 

CPIF_D 

578.116 

1846.206 

.313 

FFP_D 

1606.234 

2039.100 

.788 

FPIF_D 

-1001.003 

1904.692 

-.526 

CPAF_D 

-1127.917 

1872.168 

-.602 

CPFF_D 

-2009.671 

1901.696 

-1.057 

AcqType_Mod_D 

1591.428 

927.534 

1.716 

AcqSegment_C4ISR_D 

-16412.003 

1844.091 

-8.900 

AcqSegment_Missiles_D 

-15970.579 

1799.126 

-8.877 

AcqSegment_Aircraft_D 

-16507.851 

1732.202 

-9.530 

Program_Fractional_Year 

170.069 

77.163 

2.204 

ContractEACLessCurrentContractPrice 

.673 

1.348 

.499 

Contract  Schedule  Variance 

-55.089 

20.052 

-2.747 

Dependent  Variable:  CurrentLessInitialProgramCost 

R2  =  .565 

N  =  409 

This  final  regression,  which  includes  Contract  Schedule  Variance  and  switches 
from  CurrentEACLessInitialContractPrice  to  CurrentEACLessCurrentContractPrice 
shows  that  as  expected  contract  price  overruns  (ContractrEACLessCurrentContractPrice) 
increase,  program  cost  growth  increases.  This  logic  is  shown  visually  in  Figure  14.  The 
variable  is  not  statistically  significant,  but  was  retained  in  this  exploratory  analysis  The 
sign  of  the  contract  schedule  variance  for  the  current  year  was  negative,  which  was 
expected  since  negative  contract  variances  indicate  poor  outcomes.  Contract  Schedule 
Variance  was  also  significant;  thus,  a  poor  contract  schedule  variance  outcome  was 
significant  and,  therefore,  explained  significant  variation  in  program  cost  growth.  In 
other  words,  an  increase  in  Contract  Schedule  Variance  (a  good  outcome)  can  be 
expected  to  reduce  the  final  cost  of  a  program  (also  a  good  outcome).  The  fact  that  the 
inclusion  of  Contract  Schedule  Variance  caused  the  amended  contract  price  variable  to 
switch  signs,  and  lose  significance,  indicates  that  the  correlations  among  the  contract 
level  variables  is  sufficiently  complex  that  additional  work  on  the  model  specification  is 
required. 
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Figure  14.  Typical  Program  and  Contract  Cost  Growth  Over  Time 

Figure  14,  a  notional  figure,  shows  contract  EAC  growth  summing  to  the  growth 
in  the  program  SAR  current  estimate.  This  ignores  the  fact  that  programs  contain 
significant  government  direct  expenditures  that  do  not  flow  through  SAR-reportable 
contracts.  These  government  direct  expenditures  come  from  DoD  work  that 
complements  the  contracted  efforts,  but  is  not  listed  in  the  SAR.  The  regression  in  Table 
13  follows  the  expected  relationship  shown  in  Figure  14  between  contract  price  growth 
and  program  cost  growth. 

2.  Regressions  of  Program  and  Contract  Variances 

We  focused  this  portion  of  our  modeling  efforts  on  relating  cost,  schedule,  and 
engineering  variances  at  the  program  level  and  earned  value  cost  and  schedule  variances 
at  the  contract  level.  To  thoroughly  examine  the  effect  of  variables  at  both  the  program 
and  the  contract  level,  we  ran  regressions  in  an  iterative  manner,  building  to  combined 


program  and  contract  variance  regressions.  First,  however,  we  display  a  correlation  table 
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that  contains  the  simple  correlation  coefficients  among  the  program  and  contract 
variances. 

We  are  reminded  that  negative  contract  cost  and  schedule  variances  reflect  a  bad 
outcome  and  negative  program  variances  for  cost  and  schedule  reflect  good  outcomes.  A 
positive  sign  in  engineering  variance  is  acceptable  if  there  is  a  valid  rise  in  the  military 
requirements  during  the  program,  which  EV  captures.  The  correlations  between  program 
and  contract  variances  are  shown  in  Table  14  and  significant  correlations  are  highlighted. 


Table  14.  Program  and  Contract  Variance  Correlations 

Correlations 


Program 

Schedule 

Variances 

Program 

Engineering 

Variances 

Program  Cost 

Variance 

Contract  Cost 

Variance 

Contract 

Schedule 

Variance 

Program 

Pearson 

1 

-.008 

.245** 

-.241** 

-.280** 

Schedule 

Correlation 

Variances 

Sig.  (2-tailed) 

.801 

.000 

.000 

.000 

N 

904 

904 

904 

441 

441 

Program 

Pearson 

-.008 

1 

.301** 

-.147** 

.071 

Engineerin 

Correlation 

g 

Sig.  (2-tailed) 

.801 

.000 

.002 

.137 

Variances 

N 

904 

904 

904 

441 

441 

Program 

Pearson 

.245** 

.301** 

1 

-.107* 

-.090 

Cost 

Correlation 

Variance 

Sig.  (2-tailed) 

.000 

.000 

.024 

.058 

N 

904 

904 

904 

441 

441 

Contract 

Pearson 

-.241** 

-.147** 

-.107* 

1 

.224** 

Cost 

Correlation 

Variance 

Sig.  (2-tailed) 

.000 

.002 

.024 

.000 

N 

441 

441 

441 

441 

441 

Contract 

Pearson 

-.280** 

.071 

-.090 

.224** 

1 

Schedule 

Correlation 

Variance 

Sig.  (2-tailed) 

.000 

.137 

.058 

.000 

N 

441 

441 

441 

441 

441 

**.  Correlation  is  significant  at  the  0.01  level  (2-tailed). 
*.  Correlation  is  significant  at  the  0.05  level  (2-tailed). 
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The  correlation  table  captures  the  size  and  significance  of  the  correlations 
between  each  of  the  variances  under  investigation.  Of  particular  interest  are  the 
significant  correlations  between  program-level  variances  and  contract-level  variances. 
Program  schedule  variance  is  significantly  related  to  both  contract  cost  variance  and 
contract  schedule  variance.  Program  engineering  variance  is  significantly  associated 
with  contract  cost  variance,  and  program  cost  variance  is  significantly  associated  with 
contract  cost  variance.  While  this  table  indicates  that  there  are  connections  among  key 
variances,  what  is  needed  are  models  that  isolate  the  variance  connections,  holding  other 
variables  constant.  Therefore,  regression  analysis  is  required.  Attention  will  be  focused 
on  regression  equations  in  which  relevant  cost  variance  is  the  dependent  variable.  Other 
models  might  be  considered  in  which  program  schedule  variance,  program  engineering 
variance,  and  contract  schedule  variance  are  dependent  variables,  but  this  is  beyond  the 
scope  of  this  research  and  awaits  further  analysis. 

First,  a  program  variances-only  regression  was  constructed,  followed  by  a 
contract  earned  value  variances  regression.  These  variance  regressions  were  used  to 
inform  the  selection  of  control  variables  for  our  combined  program  and  contract 
variances  regression.  The  intent  of  the  last  regression  was  to  establish  if  there  is  a  direct 
relationship  between  the  variance  metrics  used  to  track  programs  and  the  variance  metrics 
used  to  track  contract  performance. 

The  first  regression  examined  was  for  program  cost  variance.  Program  cost 
variance  is  composed  of  multiple  components.  In  this  research  we  focused  in  part  on  the 
relationship  of  program  cost  variance  to  program  schedule  (SY)  and  program  engineering 
variance  (EV).  Since  cost  variance  is  composed  of  SV,  EV,  and  additional  categories  of 
cost  variance,  it  is  inappropriate  to  use  SV  and  EV  as  independent  variables,  with  cost 
variance  as  the  dependent  variable.  Doing  so  could  have  likely  inflated  the  correlation  of 
SV  and  EV  to  cost  variance.  To  resolve  this  concern,  the  program  variance  regressions 
used  cost  variance  less  schedule  and  engineering  cost  variances.  To  simplify  the 
discussion,  this  is  designated  as  Program  Net  Cost  Variance  in  the  regressions  employing 
this  variable. 
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It  should  also  be  noted  that  all  of  the  variance  measures  are  those  reported  in  the 
SAR  reports  for  the  current  period,  which  is  frequently  a  calendar  year.  The  information 
on  cumulative  variance  through  the  period  is  not  employed.  Also,  all  program  variance 
data  are  measured  in  FY10  millions  of  dollars. 

The  initial  regression  in  which  Program  Net  Cost  Variance  is  the  dependent 
variable,  regressed  on  the  explanatory  variables,  is  shown  in  Table  15.  The  acquisition 
segment  dummy  variables,  the  modification  dummy  variable,  the  EMD-period  dummy 
variable,  the  year  in  which  MS  B  was  achieved,  and  the  time  since  program  establishment 
were  included  to  isolate  the  effect  of  SV  and  EV  on  Program  Net  Cost  Variance. 
Contract  type  dummy  variables  and  EMD  contract  type  dummy  variables  were  also  used 
to  control  for  contract  type  effects.  Each  contract  type  dummy  variable  had  an  EMD 
contract  type  dummy  variable  associated  with  it,  except  for  FPIF,  which  was  never  the 
predominant  contract  type  for  an  EMD  contract.  The  EMD  contract  type  dummy 
variables,  combined  with  contract  type  variables  not  tied  to  EMD,  estimate  the  effect  of 
contract  type  during  EMD.  Therefore,  the  effect  of  a  contract  type  dummy  variable, 
which  is  not  identified  with  EMD,  should,  when  EMD  is  0,  identify  the  effect  of 
predominant  contract  type  during  production. 
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Table  15.  Initial  Regression  of  Program  Cost  Variances 


Model 

Coefficient 

Std.  Error 

f-statistic 

(Constant) 

-34602.05 

38478.021 

-0.899 

AcqSegment  Aircraft  D 

-307.233 

268.769 

-1.143 

AcqSegment C4ISR D 

-320.491 

282.355 

-1.135 

AcqSegment  Missiles  D 

-299.473 

269.202 

-1.112 

AcqType Mod D 

239.014 

175.469 

1.362 

EMDBCpointdate  D 

-282.386 

527.93 

-0.535 

MilestoneBscheduleachieved Fractional Year 

17.612 

19.197 

0.917 

Program Fractional Year 

3.425 

18.647 

0.184 

CPIF EMD CaseContractType D 

31.929 

605.822 

0.053 

F  F  P E  M  D CaseContractT  y  pe D 

82.528 

800.656 

0.103 

CPAF EMD CaseContractType D 

-217.799 

597.086 

-0.365 

CPFF EMD CaseContractType D 

-478.406 

731.539 

-0.654 

CPIFD 

-70.673 

319.889 

-0.221 

FFPD 

-245.879 

288.641 

-0.852 

FPIFD 

-579.157 

336.471 

-1.721 

CPAFD 

-203.669 

325.312 

-0.626 

CPFFD 

508.229 

346.098 

1.468 

Program  Schedule  Variance 

1.13 

0.307 

3.679 

Program  Engineering  Variance 

1.059 

0.202 

5.235 

Dependent  Variable:  Program  Net  Cost  Variance,  Constant2010$M 

R2  =  .086 

N  =  902 

The  very  low  /-statistics  found  for  the  Contract  Type  EMD  Case  Contract  Type 
dummy  variables  indicated  that  these  variables  should  be  eliminated  unless  inserted  as  a 
control  variable  from  the  final  regression.  Subsequent  regressions  (not  shown  in  this 
research)  also  revealed  that  Program_Fractional_Year  (the  time  since  program  initiation) 
had  a  non-significant  /-statistic;  thus,  ProgramFractionalYear  was  removed.  The 
variables  in  the  final  program  cost  variance  regression,  shown  in  Table  16,  are  otherwise 
the  same  as  those  shown  in  the  initial  program  cost  variance  regression. 
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Table  16.  Final  Regression  of  Program  Cost  Variances 


Model 

Coefficient 

Std.  Error 

f-statistic 

(Constant) 

-28472.465 

29499.223 

-0.965 

AcqSegment Aircraft D 

-342.875 

262.09 

-1.308 

AcqSegment C4ISR D 

-330.648 

277.858 

-1.19 

AcqSegment  Missiles  D 

-328.202 

264.874 

-1.239 

AcqType Mod D 

268.756 

171.754 

1.565 

EMDBCpointdate D 

-420.883 

180.665 

-2.33 

MilestoneBscheduleachieved Fractional Year 

14.591 

14.761 

0.988 

CPIFD 

-78.577 

273.31 

-0.288 

FFPD 

-281.199 

252.884 

-1.112 

FPIF D 

-625.172 

305.184 

-2.049 

CPAFD 

-281.118 

267.87 

-1.049 

CPFFD 

404.442 

298.584 

1.355 

Program  Schedule  Variance 

1.13 

0.306 

3.687 

Program  Engineering  Variance 

1.064 

0.198 

5.378 

Dependent  Variable:  Program  Net  Cost  Variance,  Constant2010$M 

R2  =  .084 

N  =  902 

In  the  final  model,  the  R  dropped  slightly,  but  the  remaining  /-statistics  improved 
enough  to  justify  eliminating  the  EMD  contract  type  dummy  variables.  The  program 
segment  variables  are  not  statistically  different  from  the  reference  Ships  binary  variables. 
The  Milestone  B  variable  is  retained  for  future  consideration  in  a  re-specified  model,  and 
only  the  FPIF  binary  variable  is  significantly  lower  than  the  reference  Hybrid  contract 
variable.  Notice  that  the  largest  negative  effect  occurred  when  FPIF  was  the  predominant 
contract  type.  However,  because  FPIF  was  never  a  predominant  contact  type  during 
EMD,  this  result  cannot  be  applied  to  a  hypothetical  situation  in  which  an  FPIF  contract 
was  the  predominant  contract  type  during  the  EMD  phase.  The  positively  significant 
effect  of  program  schedule  variance  and  program  engineering  variance  on  Program  Net 
Cost  Variance  (program  cost  variance  less  schedule  and  engineering  variance)  is  an 
important  finding.  The  coefficients  of  these  two  variables  are  close  to  1,  so  that,  other 
things  being  equal,  a  one  dollar  increase  in  each  of  these  variables  increases  Program  Net 
Cost  Variance  by  about  one  dollar. 


The  next  regression  examined  was  for  contract  earned  value  variances.  This 
regression  sought  to  better  understand  the  effect  of  contract  schedule  variance  on  contract 
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cost  variance.  The  initial  regression  controlled  for  contract  type  using  contract  type 
dummy  variables,  including  contract  dummy  variables  designed  to  estimate  the  effect 
found  during  EMD.  The  initial  regression  of  contract  cost  variances  on  the  selected 
explanatory  variables  is  shown  in  Table  17. 


Table  17.  Initial  Regression  of  Contract  Variances 


Model 

Coefficient 

Std.  Error 

f-statistic 

(Constant) 

0.531 

9.482 

0.056 

CPIF EMD CaseContractType D 

3.014 

9.549 

0.316 

F  F  PE  M  DCaseContractT  ype D 

-0.874 

46.412 

-0.019 

CPAF EMD CaseContractType D 

-4.001 

8.481 

-0.472 

CPFF EMD CaseContractType D 

3.685 

16.496 

0.223 

CPIFD 

-7.03 

10.593 

-0.664 

FFPD 

0.343 

13.27 

0.026 

FPIFD 

4.201 

10.933 

0.384 

CPAF D 

-7.461 

10.526 

-0.709 

CPFFD 

-7.733 

1 1 .499 

-0.672 

Contract  Schedule  Variance 

0.797 

0.164 

4.867 

Dependent  Variable:  Contract  Cost  Variance 

R2  =  .061 

N  =  441 

We  found  that  increases  in  contract  schedule  variance  have  a  significant  positive 
effect  on  contract  cost  variance.  Contract  type,  whether  interacting  or  not  interacting  with 
EMD,  does  not  have  a  significant  effect  on  contract  cost  variance  relative  to  the  reference 
contract. 

After  examining  multiple  iterations  of  the  model,  we  examined  the  final 
regression  of  contract  cost  variance.  This  revised  model  eliminated  the  dummy  variables 
in  which  EMD  is  tied  to  contract  type  because  of  poor  /-statistics.  After  revising  the 
initial  hypothesis  and  specifying  that  acquisition  segment  dummy  variables,  the 
modification  acquisition  type,  date  at  which  MS  B  was  achieved,  and  the  years  since 
MDAP  inception  belong  in  the  model,  we  obtained  the  results  shown  in  Table  18. 
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Table  18.  Final  Regression  of  Contract  Cost  Variance 


Model 

Coefficient 

Std.  Error 

f-statistic 

(Constant) 

-1141.612 

1444.43 

-0.79 

AcqSegment Aircraft D 

14.983 

8.857 

1.692 

AcqSegment  C4ISR  D 

26.301 

9.484 

2.773 

AcqSegment Missiles D 

24.144 

9.246 

2.611 

AcqType Mod D 

8.303 

7.214 

1.151 

MilestoneBscheduleachieved Fractional Year 

0.555 

0.721 

0.769 

Program Fractional Year 

1.142 

0.678 

1.685 

CPIFD 

-2.89 

10.417 

-0.277 

FFPD 

4.183 

13.492 

0.31 

FPIFD 

11.941 

11.369 

1.05 

CPAFD 

-4.963 

10.442 

-0.475 

CPFFD 

0.437 

11.376 

0.038 

Contract  Schedule  Variance 

0.733 

0.163 

4.495 

Dependent  Variable:  Contract  Cost  Variance 

R2  =  .086 

N  =  441 

Many  of  the  additional  variables  included  are  statistically  significant.  However, 
the  predominant  contract  type  variables  continue  to  be  statistically  non-significant 
compared  with  the  reference  Hybrid  contract.  More  important,  the  significant  positive 
relationship  between  contract  schedule  variance  and  earned  value  cost  variance  continues 
to  apply.  This  means  that  an  increase  in  budgeted  cost  of  work  performed  less  budgeted 
cost  of  work  scheduled  (a  positive  outcome)  is  associated  with  an  increase  in  budgeted 
cost  of  work  performed  less  actual  cost  of  work  performed  (also  a  positive  outcome). 

After  conducting  program  variance-only  and  contract  variance-only  regressions, 
we  examined  the  regressions  shown  in  Table  19  with  both  program  and  contract 
variances.  The  initial  regression  sought  to  establish  the  effect  of  the  contract  cost 
variance,  contract  schedule  variance,  program  engineering  variance,  and  program 
schedule  variance  on  Program  Net  Cost  Variance.  Additional  variables  were  added  to 
control  for  the  effects  of  being  in  EMD  and  for  the  predominant  contract  type  by  program 
during  EMD. 
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Table  19.  Initial  Regression  of  Program  and  Conf 

:ract  Variances 

Model 

Coefficient 

Std.  Error 

^-statistic 

(Constant) 

-38.77 

-0.273 

Program  Engineering  Variance  Constant2010$M 

1.084 

0.186 

3.997 

Program  Schedule  Variance  Constant2010$M 

1.729 

0.232 

4.771 

EMDBCpointdate D 

-297.978 

-0.071 

-1.463 

EMDCPIFD 

692.664 

0.169 

3.064 

EMDCPAFD 

96.755 

0.027 

0.486 

EMDCPFFD 

405.657 

0.08 

1.579 

Contract  Cost  Variance 

0.577 

0.016 

0.327 

Contract  Schedule  Variance 

-3.517 

-0.028 

-0.572 

Dependent  Variable:  Program  Net  Cost  Variance  Constant2010$M 

R2  =  .104 

N  =  439 

With  the  program  engineering  and  schedule  variances  included  in  the  regression, 
the  contract  cost  and  contract  schedule  variances  were  not  significant.  We  can  note  from 
Table  14  that  the  correlation  between  program  schedule  variance  and  both  contract  cost 
variance  and  contract  schedule  variance  were  statistically  significant,  and  the  correlation 
between  program  engineering  variance  and  contract  cost  variance  was  statistically 
significant.  One  can  conjecture  that  multicollinearity  might  have  impacted  the 
significance  of  contract  variance  variables.  In  the  re-specification  of  the  model,  the 
program  variance  explanatory  variables  were  deleted,  and  additional  variables  were 
added  to  determine  whether  an  alternative  model  might  have  superior  statistical 
properties.  The  final  regression  of  program  and  contract  variances  resulting  from  these 
changes  is  shown  in  Table  20. 
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Table  20.  Final  Regression  of  Program  and  Contract  Variances 


Model 

Coefficient 

Std.  Error 

f-statistic 

(Constant) 

31915.502 

58874.493 

0.542 

AcqSegment Aircraft D 

150.025 

365.233 

0.411 

AcqSegment C4ISR D 

-296.542 

391.406 

-0.758 

AcqSegment  Missiles  D 

-249.867 

375.703 

-0.665 

AcqType Mod D 

-29.488 

304.061 

-0.097 

EMDBCpointdate D 

71.881 

827.471 

0.087 

MilestoneBscheduleachieved Fractional Year 

-15.458 

29.383 

-0.526 

Program Fractional Year 

-63.91 

30.814 

-2.074 

CPIF EMD CaseContractType D 

-743.446 

894.203 

-0.831 

FFP EMD CaseContractType D 

1244.68 

2049.642 

0.607 

CPAF EM  D CaseContractT  ype D 

-1061.793 

903.484 

-1.175 

CPFF EMD CaseContractType D 

-1462.308 

1079.94 

-1.354 

CPIFD 

132.394 

509.539 

0.26 

FFPD 

34.286 

604.259 

0.057 

FPIF D 

-238.723 

532.09 

-0.449 

CPAFD 

68.264 

520.106 

0.131 

CPFFD 

517.904 

541.03 

0.957 

Contract  Cost  Variance 

-3.281 

1.939 

-1.692 

Contract  Schedule  Variance 

-10.073 

6.722 

-1.499 

Dependent  Variable:  SubtotalCurrentChangesConstant2010$M 

R2  =  .062 

N  =  439 

The  newly  specified  regression,  which  now  includes  (total)  program  cost 
variance,  significantly  enhances  the  statistical  significance  of  the  contract  variances  and 
yields  coefficient  signs  that  are  expected;  increasing  contract  cost  variance  and  contract 
schedule  variance  is  correlated  with  decreasing  program  cost  variances.  This  negative 
correlation  actually  indicates  that  contract  and  program  variances  are  traveling  in  the 
same  direction,  since  the  sign  of  a  good  outcome  for  contracts  is  positive,  while  the  sign 
of  a  good  outcome  for  programs  is  negative.  Therefore,  a  connection  between  program- 
level  data  and  contract-level  data  was  identified.  In  addition,  program  fractional  year  was 
a  statistically  significant  variable  in  this  revised  model. 

The  understanding  of  program  and  contract  variances  we  built  using  multiple 
regression  analysis  supports  our  model  of  the  relationship  between  contract  price  growth 
and  program  cost  growth,  shown  in  Figure  15. 
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Variables  affectin 


Contract  Cost: 


(No  exclusive 
variables  discovered 
for  this  relationshint 


Variables  affectin 


Program  and 


Contract  Cost: 


Contract  Schedule 
Variance 


Contract  Measures: 


Contract  EAC  Less 
Initial  (or  Current) 
Contract  Price 
Contract  Cost  Variance 


Variables  affecting 

\ 

Program  Measures: 

Program  Cost: 

\ 

Current  Less  Initial 

Program  Schedule 

Program  Cost 

Variance 

Program  Net  Cost 

Program  Engineering 

Variance 

Figure  15.  Relationship  Between  Variables 

The  relationships  shown  in  Figure  15  could  form  the  starting  point  for  further 
analysis  of  the  relationship  between  contract  price  growth  and  program  cost  growth. 
Chapter  VI  includes  suggestions  regarding  future  analysis.  Of  particular  interest  is 
structural  equation  modeling,  which  offers  the  possibility  to  separate  direct  and  indirect 
effects  between  variables  affecting  both  contract  price  growth  and  program  cost  growth. 
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VI.  SUMMARY,  CONCLUSIONS,  AND  RECOMMENDATIONS 


A.  DISCUSSION  OF  RESULTS 

This  research  focused  on  quantifying  relationships  between  acquisition  program 
attributes,  choices,  and  outcomes.  The  results  recorded  in  Chapter  V,  while  exploratory 
in  nature,  show  the  relationships  that  we  found  in  attempting  to  address  each  of  our 
research  questions.  In  this  chapter,  we  review  those  results,  identity  limitations  of  the 
study,  and  present  recommendations  for  further  research. 

Our  results  do  not  provide  a  definitive  conclusion  to  all  of  our  research  questions, 
but  they  do  offer  insights  regarding  the  relationships  between  program  and  contract 
variances,  along  with  contract  types.  As  discussed  in  this  study,  one  area  of  potential 
concern  is  that  the  use  of  fixed-price  contracts  during  R&D  may  limit  contractors’  efforts 
to  identify  possible  technical  risks  early  in  a  program  that  would  prevent  future  system 
design  problems.  There  were  only  3 1  fixed-price  contracts  started  during  EMD,  and  no 
program  had  predominant  fixed-price  contracts  during  the  EMD  phase.  In  this 
exploratory  analysis,  the  three  programs  with  predominantly  fixed-price  contracts 
initiated  under  RDT&E  appropriations  performed  unfavorably  on  program  costs 
compared  to  other  contract  types  in  the  dataset.  This  may  be  attributed  to  the 
establishment  of  a  fixed-price  contract  amount.  The  CY,  SY,  and  EV  all  performed 
worse  for  programs  with  fixed-price  RDT&E,  when  compared  to  the  remaining  MDAPs. 
This  could  be  a  result  of  the  increased  uncertainty  and  technical  risk  experienced  during 
RDT&E.  Programs  with  a  majority  of  contract  spending  on  fixed-price  contracts  during 
the  development  phase  may  not  be  able  to  appropriately  hedge  technical  risk.  This  might 
be  because  the  ceiling  price  could  provide  incentives  for  a  contract  change  to  be 
approved.  Although  not  conclusive,  the  use  of  incentives  and  award  fees  may  run  higher 
costs  and  longer  schedules,  but  could  prevent  expensive  future  problem  identification. 

When  comparing  the  different  acquisition  segment  results,  C4ISR  performed  the 
least  favorably,  with  the  highest  schedule  growth  of  13%  and  the  second  highest  cost 
growth  of  10%.  The  Aircraft  segment  had  the  highest  cost  growth,  just  over  10%,  but  the 
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lowest  schedule  growth,  1.5%.  The  Missiles,  weapons,  and  ammunition  segment  fell  into 
the  middle  of  the  three  categories;  the  single  submarine  program  we  analyzed  was  not 
comparable  to  the  other  segments. 

Our  findings  regarding  CV,  SV,  and  EV  as  leading  indicators  of  later  period  CV, 
SV,  and  EV  were  only  slightly  conclusive,  in  part  due  to  SV  and  EV  not  being 
extensively  studied  as  dependent  variables.  Our  findings  regarding  CV  were  partially 
conclusive.  We  found  that  an  increase  in  program  year  (the  number  of  years  since 
program  inception  at  MS  B)  was  significantly  positively  correlated  with  an  increase  in 
contract  price  growth  (Table  11)  and  program  cost  growth  (Tables  9  and  13).  Also,  an 
increase  in  program  year  was  significantly  negatively  correlated  with  an  increase  in 
program  cost  variance  (Table  20).  Program  cost  growth  is  a  measure  of  scope  increases 
and  overruns,  while  program  cost  variance  is  a  measure  of  changes  during  the  past  year  to 
a  program’s  expected  cost.  The  fact  that  the  program  year  independent  variable  had 
opposite  and  significant  signs  for  the  two  measures  of  program  cost  changes  likely 
indicates  that  rebaselining  significantly  impacts  the  relationship  of  one  or  both  to 
program  year.  Program  year  had  a  significant  t-statistic  for  the  final  program  variance 
regression  (Table  20)  than  contract  cost  variance  or  contract  schedule  variance.  The 
strength  of  this  relationship  and  the  opposite  effect  for  program  growth  and  program 
variance  measures  could  indicate  that  this  variable  is  actually  capturing  the  effect  of 
rebaselining  on  program  metrics.  Rebaselining  could  be  a  significant  source  of  the 
limited  correlation  between  contract  variances  and  program  variances  in  the  program 
variances  regression.  Further  study  of  the  relationship  between  program  variances  and 
further  study  of  the  effects  of  rebaselining  could  improve  the  explanatory  power  of  our 
models. 


The  results  achieved  using  regressions  established  preliminary  findings.  The 
Aircraft,  C4ISR,  and  Missile  segments  all  showed  significantly  lower  cost  growth  than 
Ships.  Program  managers  of  ships  and  submarines  should  be  cognizant  of  the  risk  of 
higher  cost  growth  that  exists  on  already  expensive  programs.  Additionally,  regressions 
showed  modifications  actually  ran  higher  cost  growth  than  did  new  starts  and  initial 
programs.  Although  not  at  a  significant  level,  the  predominant  use  of  fixed-price 
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contracts  during  production  was  associated  with  lower  cost  growth.  This  follows  our 
discussion  of  risk  with  regard  to  contract  type  in  Chapter  II. 

We  are  one  of  the  first  research  teams  to  analyze  the  relationship  of  EVM  contract 
variances  and  program  variances.  The  connection  between  program  and  contract 
variances  identified  supports  the  view  that  contract  price  growth  affects  program  cost 
growth.  Understanding  exactly  how  program  growth  is  affected  by  contract  growth  will 
allow  program  managers  to  better  control  costs.  No  conclusive  effects  were  established, 
but  this  research  has  laid  the  ground  work  for  future  research,  such  as  SEM  (discussed  in 
Section  D),  to  refine  the  results  achieved. 

B.  RECOMMENDATIONS 

In  this  research,  we  partially  confirmed  the  reservations  of  acquisition 
professionals  who  believe  the  push  for  fixed-price  R&D  from  sources  such  as  Under 
Secretary  Carter’s  (2010)  Better  Buying  Power  may  not  always  be  the  best  contract  type. 
We  also  determined  that  programs  with  a  majority  of  fixed-price  RDT&E  contracts  tend 
to  have  lower  cost  growth  overall  and  lower  cost  growth  attributable  to  schedule  growth, 
but  higher  engineering  cost  growth  compared  to  all  MDAPs.  By  treating  engineering 
cost  growth  as  a  proxy  for  technical  changes,  and  by  considering  the  attendant  technical 
risks  associated  with  those  changes,  it  is  clear  that  fixed-price  RDT&E  saves  money  and 
time,  but  increases  technical  risk.  This  increased  technical  risk  may  be  acceptable  for 
RDT&E  spending  during  the  production  phase,  but  would  likely  be  unadvisable  during 
EMD,  when  the  costs  of  discovering  problems  in  systems  engineering  are  substantial. 

We  examined  the  use  of  fixed-price  contracts  for  RDT&E  spending  in  this 
research,  but  we  did  not  examine  the  use  of  fixed-price  contracts  during  EMD  because 
none  of  the  MDAPs  in  our  sample  had  fixed-price  contracts  as  the  majority  contract  type 
for  EMD.  However,  it  is  possible  that  fixed-price  contracts  may  be  successfully  extended 
into  EMD  for  a  limited  number  of  systems,  particularly  those  that  exceed  the  now- 
strengthened  requirements  for  minimum  TRLs.  Program  managers  should  proceed  with 
caution,  due  to  the  potential  for  increased  technical  risk.  Only  9%  of  RDT&E  contracts 
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in  our  sample  were  fixed-price,  and  the  vast  majority  of  those  were  during  the  production 
phase. 


C.  STUDY  LIMITATIONS 

The  data  we  used  in  this  study  came  directly  from  each  program’s  SAR,  including 
cost  variances  at  the  program  level  and  earned  value  at  the  contract  level.  We  analyzed  a 
relatively  small  number  of  programs  (31)  due  to  the  limited  availability  of  data  that  fit 
our  study’s  requirements  and  to  the  labor-intensive  nature  of  the  data  collection.  The 
dataset  included  multiple  years  and  contracts  for  each  program,  which  provided  us  with  a 
significant  number  of  observations. 

SAR  data  have  many  limitations  that  we  observed  during  this  study.  Although 
recent  changes  to  regulations  driven  by  WSARA  were  made  in  an  attempt  to  reduce 
rebaselining,  our  period  of  study  went  back  as  far  as  1997  for  some  programs.  Due  to 
shifting  rebaselining  policy  over  time,  it  was  difficult  to  determine  the  decisions  made 
and  the  effects  of  these  decisions  on  variances  after  a  program  was  rebaselined.  After 
reviewing  multiple  SAR  variances  at  the  program  and  contract  level,  there  were  missing 
explanations  as  to  the  cause  of  variances  required.  Multiple  SARs  had  vague  information 
regarding  contract  type.  It  appeared  that  cost  variances  were  allocated  inconsistently 
between  SAR  categories  and  between  years.  The  cost  information  provided  for  future 
years  reflected  projected  budget  values  and  was  not  always  consistent  with  cost 
estimates.  Multiple  contract  types  were  listed  with  no  identification  of  dollar  amounts  or 
designation  of  dominant  type. 

Even  with  the  limitations  in  the  SARs,  we  were  able  to  build  an  extensive  dataset. 
The  data  also  had  their  own  limitations.  The  low  number  of  fixed-price  contracts  utilized 
during  the  EMD  phase  prevented  us  from  producing  thorough  results  of  the  effects  of 
fixed-price  development  contracts  on  program  variances.  Due  to  a  lack  of  variability  in 
predominant  contract  type  during  the  EMD  phase  and  due  to  no  program  having  a 
predominant  fixed-price  contract  type,  we  removed  those  programs  with  predominant 
RDT&E  contract  type  to  compare  differences  in  production  contract  type.  Also,  as  we 
have  indicated,  the  regression  analysis  must  be  viewed  as  exploratory.  The  models 
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estimated  are  a  step  on  the  road  to  hypothesizing  causal  models  that  may  succeed  in 
obtaining  definitive  findings. 

D.  AREAS  FOR  FUTURE  RESEARCH 

1.  Structural  Equation  Modeling 

In  addition  to  looking  at  definable,  quantitative  cost  growth,  it  would  also  be 
helpful  to  examine  root  sources  of  variance  within  programs.  In  Chapter  II,  we  discussed 
risk  in  different  acquisition  phases.  Risk  in  acquisition  programs  is  an  undefined 
characteristic  that  program  managers  continually  try  to  control.  Programs  have 
designated  risk  reserves,  but  these  reserves  and  levels  of  uncertainty  or  risk  are  not 
reportable  on  SARs.  Although  risks  are  understood  in  the  acquisition  community,  they 
could  have  direct  effects  on  program  or  contract  variances. 

Program  risk  can  be  characterized  by  three  subparts:  cost,  schedule,  and  technical 
risk.  Once  these  risks  are  recognized,  the  result  could  include  a  change  in  price, 
schedule,  or  technology  that  could  ultimately  correspond  to  a  program  CV,  SV,  or  EV. 
Any  single  change  could  lead  to  changes  throughout  a  program.  For  example,  funding 
from  Congress  could  be  reduced,  affecting  the  quantity  of  affordable  systems,  adjusting 
the  design,  and  lengthening  the  schedule. 

To  allow  for  the  more  advanced  statistical  methods  of  cross-sectional,  time-series 
analysis,  we  recommend  use  of  structural  equation  modeling  (SEM)  to  analyze  program 
and  contract  variance  data.  A  thorough  investigation  should  be  conducted,  including  an 
investigation  of  the  use  of  exploratory  factor  analysis  to  identify  latent  variables,  and  the 
results  of  this  investigation  should  be  used  to  assess  the  applicability  of  SEM  to  program 
and  contract  variance  data.  SEM  differs  from  the  prior  techniques  in  its  “ability  to 
simultaneously  estimate  multiple  dependence  relationships  ...  while  also  incorporating 
multiple  measures  for  each  [construct]”  (Hair  et  al.,  2009,  p.  609).  Constructs  would  be 
the  dependent  and  independent  variables  included  in  the  analysis.  SEM  examines  both 
interdependent  and  dependent  relationships  simultaneously  in  a  manner  akin  to 
combining  factor  analysis  and  multiple  regression  analysis  (Hair  et  al.,  2009). 
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SEM  models  also  permit  the  simultaneous  variance  of  multiple  factors  including 
risk.  This  method  allows  the  model  to  include  conceptually  constructed  variables  that 
underlie  the  measurable  variables  and  could  help  explain  the  full  effect  of  fixed-price 
incentive  R&D  contracts  within  MDAPs  on  later  period  cost  and  schedule  variance 
during  the  production  and  deployment  phases.  The  use  of  SEM  could  allow  for  the 
examination  of  variance  of  multiple,  interrelated,  simultaneously  varying  factors,  such  as 
the  cost,  schedule,  and  engineering  variances  of  a  program  and  the  earned  value  cost  and 
schedule  variances  of  that  program’s  contracts.  At  the  same  time,  it  incorporates 
unobserved  variables  that  could  be  categorized  as  programmatic  risk  and  its  various  risk 
components.  These  risk  measures  would  mediate  the  relationship  between  contract  type 
and  other  variables,  and  the  program’s  cost  objectives,  and  thereby  clarify  the  effect  of 
contract  type  on  the  desired  outcome.  The  objective  of  this  future  research  would  be  to 
determine  the  effects  of  the  increased  usage  of  fixed-price  R&D  contracts  during  EMD 
on  long-term  program  CV,  SV,  and  EV.  It  would  also  look  to  determine  whether 
preproduction  CV,  SV,  and  EV,  and  the  associated  contract  variance  measures,  are 
indicators  of  CV,  SV,  and/or  EV  and  contract  performance  during  production. 

2.  Normality 

Unlike  regression  analysis,  normality  of  the  variables  is  a  more  important 
assumption  when  SEM  is  employed.  Therefore,  another  area  deserving  future  research  is 
the  source  of  non-normality  in  program  cost  variance.  This  topic  would  lend  itself  to 
research  based  on  frank  discussions  with  past  program  managers  about  information 
hiding  and  moral  hazard  in  the  principal-agent  relationship  between  Congress  and 
program  managers.  It  would  also  lend  itself  to  further  exploration  of  the  characteristics 
of  programs  that  exhibit  drastic  deviations  from  normality. 


F.  SUMMARY 


In  this  research  we  sought  to  improve  understanding  of  the  interrelationship 
between  program  and  contract  metrics  by  quantifying  the  relationships  between  program 
and  contract  outcomes.  Our  exploratory  efforts  to  explain  program  cost  increases  using 
program  and  contract  outcomes  resulted  in  the  explanation  of  56.6%  of  program  cost 
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increases.  Our  efforts  to  explain  program  cost  variance  indicated  that  only  6.2%  of 
program  cost  variance  could  be  explained.  As  indicated  previously  in  this  chapter, 
however,  a  model  of  the  small  portion  of  the  total  variance  explained  when  analyzing  a 
response  to  a  policy  change,  such  as  a  change  in  contract  type,  provided  the  policy 
change  variable  is  statistically  significant.  Nevertheless,  additional  modeling  beyond  the 
exploratory  effort  undertaken  here  could  increase  the  ability  of  both  models  to  explain 
overall  program  outcomes,  and  how  these  change  when  a  particular  policy  variable 
changes.  This  research  also  attempted  to  show  how  preproduction  CV,  SV,  and  EV  data 
within  acquisition  programs  affect  future  variances.  The  results  were  not  significantly 
quantifiable,  and  additional  research  is  recommended.  Our  results  indicated  several 
possible  areas  of  future  research,  including  the  non-normality  in  SAR  measures  and 
SEM.  Both  areas  offer  the  possibility  of  increasing  the  effectiveness  of  the  models 
explored  in  this  project. 
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APPENDIX  A.  LIST  OF  VARIABLES  FOR  ANALYSIS 


Variables  used  in  the  research  write-up 

Name 

Explanation 

AcqSegment Aircraft D 

1,  if  Acquisition  Segment  is  Aircraft;  0,  otherwise 

AcqSegment C4ISR D 

1,  if  Acquisition  Segment  is  C4ISR;  0,  otherwise 

AcqSegment Missiles D 

1,  if  Acquisition  Segment  is  Missiles;  0,  otherwise 

AcqSegment Ships D 

1,  if  Acquisition  Segment  is  Ships;  0,  otherwise 

AcqType Mod D 

1,  if  program  Acquisition  Type  is  a  modification;  0,  otherwise 

AcqType New D 

1,  if  program  Acquisition  Type  is  a  new  start;  0,  otherwise 

CECostVarianceConstant2010$M 

Current  Program  Cost  Estimate  through  the  end  of  production  (in 
2010  constant  millions  of  dollars);  sum  of  prior  and  current  cost 
estimate  changes  and  baseline  cost  estimate 

CONTOTH D 

1,  if  contract  type  is  Hybrid/Other;  0,  otherwise 

CONTOTH_EMD_CaseContractTy 
pe D 

COMPUTE 

CONTOTH_EMD_CaseContractType_D=EMDBCpointdate_d  * 
CONTOTH D 

Contract_BasicContractType_CP_ 

D 

1,  if  CP  is  Basic  Contract  Type  for  an  individual  contract ;  0, 
otherwise  (FP,  fixed  price;  or  CP,  cost  plus) 

Contract_BasicContractType_FP_ 

D 

1,  if  FP  is  Basic  Contract  Type  for  an  individual  contract ;  0, 
otherwise  (FP,  fixed  price;  or  CP,  cost  plus) 

ContractEACLessCurrentContract 

Price 

COMPUTE 

ContractEACLessCurrentContractPrice=ProgramManagerEstimated 
PriceAtCompletion$M  -  TargetCurrentContractPrice$M 

ContractEACLessInitialContractPri 

ce 

COMPUTE 

ContractEACLesslnitialContractPrice=ProgramManagerEstimatedPri 
ceAtCompletion$M  -  TargetlnitialContractPrice$M 

ContractID 

Contract  ID  associated  with  a  contract  at  the  program  level  (NOTE: 
this  Contract  ID  repeats  for  each  program;  e.g.  each  program  has  a 
contract  "1.") 

ContractName 

Contract  Name 

ContractNumber 

Contract  Number  (obtained  from  SAR  and  corrected,  as  necessary, 
using  FPDS) 

Contractor 

Contractor  Name 

ContractType 

Contract  Type  from  SAR,  adjusted  using  FPDS  data 

Contract  Cost  Variance 

Current  Cost  Variances  (in  millions  of  dollars) 

Contract  Schedule  Variance 

Current  Schedule  Variances  (in  millions  of  dollars) 

CPAF D 

1,  if  contract  type  is  CPAF;  0,  otherwise 

CPAF EMD CaseContractType D 

COMPUTE  CPAF_EMD_CaseContractType_D=EMDBCpointdate_d  * 
CPAF D 

CPFF D 

1,  if  contract  type  is  CPFF;  0,  otherwise 

CPFF EMD CaseContractType D 

COMPUTE  CPFF_EMD_CaseContractType_D=EMDBCpointdate_d  * 
CPFF D 

CPIF_D 

1,  if  contract  type  is  CPIF;  0,  otherwise 
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CPIF EMD CaseContractType D 

COMPUTE  CPIF_EMD_CaseContractType_D=EMDBCpointdate_d  * 
CPIF D 

CurrentLessInitialContractPrice 

COMPUTE 

CurrentLesslnitialContractPrice=TargetCurrentContractPrice$M  - 
TargetlnitialContractPrice$M 

CurrentLessInitialProgramCost 

COMPUTE 

CurrentLesslnitialProgramCost=CECostVarianceConstant2010$M  - 
SARBaselineProdEstConstant2010$M 

EMD BasicContractType CP D 

1,  if  CP  predominant  contract  type  during  EMD;  0,  otherwise  (CP, 

Cost  Plus;  or  FP,  Fixed  Price) 

EMD BasicContractType FP D 

1,  if  FP  predominant  contract  type  during  EMD;  0,  otherwise  (CP, 

Cost  Plus;  or  FP,  Fixed  Price) 

EMD CONTOTH D 

1,  if  predominant  contract  type  during  EMD  =  Hybrid/Other;  0, 
otherwise 

EMD CPAF D 

1,  if  predominant  contract  type  during  EMD  =  CPAF;  0,  otherwise 

EMD CPFF D 

1,  if  predominant  contract  type  during  EMD  =  CPFF;  0,  otherwise 

EMD CPIF D 

1,  if  predominant  contract  type  during  EMD  =  CPIF;  0,  otherwise 

EMD FFP D 

1,  if  predominant  contract  type  during  EMD  =  FFP;  0,  otherwise 

EMD FPIF D 

1,  if  predominant  contract  type  during  EMD  =  FPIF;  0,  otherwise 

EMD D 

1,  if  data  represents  EMD  period;  0,  if  data  represents  production 
period 

EMDpredominantBasicContractT 

Predominant  contract  type  during  EMD  (CP,  Cost  Plus;  or  FP,  Fixed 

ype 

Price) 

FFP D 

1,  if  contract  type  is  FFP;  0,  otherwise 

FFP EMD CaseContractType D 

COMPUTE  FFP_EMD_CaseContractType_D=EMDBCpointdate_d  * 

FFP D 

FPIF D 

1,  if  contract  type  is  FPIF;  0,  otherwise 

FPIF EMD CaseContractType D 

COMPUTE  FPIF_EMD_CaseContractType_D=EMDBCpointdate_d  * 

FPI  F D 

MilestoneBscheduleachieved_Fra 

Sum  of  MilestoneBscheduleachieved_Year  + 
MilestoneBscheduleachieved_Month_12  (NOTE:  December  2010  is 

ctional Year 

then  represented  as  2011.00.) 

PROC D 

1,  if  appropriation  is  procurement;  0,  otherwise 

PROD BasicContractType CP D 

1,  if  Cost  Plus;  0,  otherwise 

PROD BasicContractType FP D 

1,  if  Fixed  Price;  0,  otherwise 

PROD CONTOTH D 

1,  if  Hybrid/Other;  0,  otherwise 

PROD CPAF D 

1,  if  CPAF;  0,  otherwise 

PROD CPFF D 

1,  if  CPFF;  0,  otherwise 

PROD CPIF D 

1,  if  CPIF;  0,  otherwise 

PROD FFP D 

1,  if  FFP;  0,  otherwise 

PROD FPIF D 

1,  if  FPIF;  0,  otherwise 

PRODpredominantBasicContractT 

Predominant  basic  contract  type  during  production  (Fixed  Price,  FP; 

ype 

or  Cost  Plus,  CP) 

PRODpredominantContractType 

Predominant  contract  type  during  production  (FFP,  FPIF,  CPIF, 
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CPAF,  CPFF,  Hybrid/Other) 

Program Fractional Year 

Years  since  program  initiation 

ProgramandContractID 

A  unique  Contract  ID  associated  with  a  contract  representing  a 
single  scope  of  work.  To  accommodate  multiple  programs  and 
contracts,  the  ID  is  structured  with  a  leading  1,  followed  by  the 
Program  ID  followed  by  a  leading  1,  followed  by  the  Contract 

ID  For  greater  than  99  programs  or  contracts,  a  leading  2 

would  be  utilized. 

Program  Engineering  Variance 
Constant2010$M 

Current  period  cost  estimate  changes  from  the  baseline  due  to 
engineering  factors  (in  2010  constant  millions  of  dollars) 

ProgramID 

ID  numbers  corresponding  to  each  program 

ProgramManagerEstimatedPrice 

AtCompletion$M 

Program  manager's  estimated  price  at  completion  (in  millions  of 
dollars) 

PROGRAMNAME 

The  name  generally  associated  with  the  program  in  recent  SARs 

Program  Net  Cost  Variance 
Constant2010$M 

SubtotalCurrentChangesBY$M  -  Program  Schedule  Variance  BY$M 
-  Program  Engineering  Variance  BY$M.  Commonly  referred  to  as 
Program  Cost  Variance  (or  just  Cost  Variance)  in  research. 

Program  Schedule  Variance 
Constant2010$M 

Current  period  cost  estimate  changes  from  the  baseline  due  to 
schedule  factors  (in  2010  constant  millions  of  dollars) 

RDTE BasicContractType CP D 

1,  if  FP  predominant  contract  type  for  RDT&E;  0,  otherwise  (CP, 

Cost  Plus;  or  FP,  Fixed  Price) 

RDTE BasicContractType FP D 

1,  if  CP  predominant  contract  type  for  RDT&E;  0,  otherwise  (CP, 

Cost  Plus;  or  FP,  Fixed  Price) 

RDTE CONTOTH D 

1,  if  predominant  contract  type  for  RDT&E  =  Hybrid/Other;  0, 
otherwise 

RDTE CPAF D 

1,  if  predominant  contract  type  for  RDT&E  =  CPAF;  0,  otherwise 

RDTE CPFF D 

1,  if  predominant  contract  type  for  RDT&E  =  CPFF;  0,  otherwise 

RDTE CPIF D 

1,  if  predominant  contract  type  for  RDT&E  =  CPIF;  0,  otherwise 

RDTE D 

1,  if  appropriation  is  RDT&E;  0,  otherwise 

RDTE FFP D 

1,  if  predominant  contract  type  for  RDT&E  =  FFP;  0,  otherwise 

RDTE FPIF D 

1,  if  predominant  contract  type  for  RDT&E  =  FPIF;  0,  otherwise 

RDTEpredominantBasicContractT 

ype 

Predominant  contract  type  for  RDT&E  contracts  (CP,  Cost  Plus;  or 

FP,  Fixed  Price) 

SAR Fractional Year 

Sum  of  SAR_Year  +  SAR_Month_12  (NOTE:  December  2010  is  then 
represented  as  2011.00.) 

SARBaselineProdEstConstant201 

0$M 

SAR  baseline  cost  estimate  through  the  end  of  production  (in  2010 
constant  millions  of  dollars) 

SubtotalCurrentChangesConstant 

2010$M 

Subtotal  of  current  period  cost  estimate  changes  from  the  baseline 
(in  2010  constant  millions  of  dollars) 

TargetCurrentContractPrice$M 

Current  Target  Contract  Price  (in  millions  of  dollars) 

TargetlnitialContractPrice$M 

Initial  Target  Contract  Price  (in  millions  of  dollars) 
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APPENDIX  B.  NORMALITY  OF  VARIANCE  DATA 


For  future  analysis,  particularly  SEM,  it  may  be  helpful  to  understand  the  extent 
to  which  the  variance  data  were  normally  distributed.  Figures  16  and  17  are  histograms 
of  program  schedule  variance  and  contract  schedule  variance,  demonstrating  this  kurtosis. 
Each  histogram  also  shows  the  expected  normal  distribution  of  the  data,  given  the  sample 
standard  deviation.  When  interpreting  these  figures  it  is  useful  to  note  that  negative 
program  variances  represent  better-than-expected  performance,  while  negative  contract 
variances  represent  poorer-than-expected  performance. 


Mean  =  16.93 
Std.  Dev.  =  140.379 
N  =  904 


Figure  16.  Program  Schedule  Variance  Histogram  With  Expected  Normal 

Distribution  Overlaid 
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Mean  =  -1.8 
Std.  Dev.  =  13.391 
N  =  441 
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Figure  17.  Contract  Schedule  Variance  Histogram  With  Expected  Normal 

Distribution  Overlaid 

There  are  two  possible  explanations  for  this  kurtosis.  The  first  is  a  failure  by 
program  managers  and  contractors  to  report  undesirable  information  due  to  improper 
incentives  corrupting  the  flow  of  information  from  agent  to  principal  at  the  program  and 
contract  levels.  The  second  explanation  is  that  the  data  represent  two  separate,  overlaid 
normal  distributions.  One  distribution  would  represent  the  typical  behavior  of  programs: 
incremental  failures  and  successes  resulting  in  relatively  small  standard  deviations  and  a 
more  compact  distribution  of  results.  The  other  distribution  would  represent  the  aberrant 
behavior  of  programs:  large  over-  or  under-estimation  in  cost  requirements,  schedule 
length,  technical  maturation,  or  integration  risk.  These  more  unusual  events  would  result 
in  larger  standard  deviations  and  a  more  “flat”  distribution  of  results.  It  is  likely  that  both 
explanations  are  partially  responsible  for  the  non-normality  present. 
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Program  managers  are  incentivized  to  report  variances  that  are  advantageous  to 
their  program.  This  leads  to  a  suppression  of  variances  representing  poorer-than- 
expected  outcomes  due  to  the  subsequent  increase  in  program  risk  increasing  the 
possibility  of  program  cancellation.  Also,  variances  representing  better-than-expected 
outcomes  are  suppressed  due  to  the  possibility  of  the  program  losing  funding,  thereby 
precluding  technical  risk  reduction.  Program  managers  are  incentivized  to  think  about 
what  is  best  for  their  program  and  Service,  while  Congress  has  different  goals.  Absent 
perfect  incentives,  differing  objectives  result  in  the  agent  performing  in  a  manner  other 
than  how  the  principle  intended  (Laffont  &  Martimort,  2002).  Information  can  be 
suppressed  by  an  agent  (the  program  manager)  to  the  extent  that  information  asymmetry 
exists  with  the  principal  (Congress).  Congress  limits  information  asymmetry  through 
statutory  requirements.  Program  managers  may  not  necessarily  be  dishonest  in  their 
reporting;  willful  blindness  in  the  form  of  excessive  pessimism  or  optimism  may  also  be 
to  blame  for  near-zero  variances.  Reporting  fewer  zero  variances  would  increase  the 
standard  deviation  of  the  sample  and  potentially  lower  the  number  of  outliers.  If  outliers 
found  in  the  data  actually  represent  past  variances  suppressed  to  zero  or  near-zero  values, 
then  the  standard  deviation  would  not  increase,  but  the  amount  of  kurtosis  would 
decrease. 

The  second  explanation,  that  the  data  represent  two  separate,  overlaid  normal 
distributions,  is  also  likely  to  contribute  to  the  kurtosis.  If  this  is  correct,  two 
distributions  are  represented  in  the  data.  One  distribution  represents  the  typical  behavior 
of  programs — incremental  failures  and  successes  resulting  in  relatively  small  standard 
deviations — and  another  distribution  represents  the  aberrant  behavior  of  programs — large 
over-  or  under-estimation  in  cost  requirements,  schedule  length,  technical  maturation,  or 
integration  risk,  resulting  in  larger  standard  deviations.  Causes  for  this  type  of  large 
program  or  contract  variance  could  be  a  large  over-  or  under-estimation  of  the  effort 
required  to  complete  a  task,  test  event  failures,  poor  estimating,  or  the  realization  of  an 
accepted  risk.  Risks  assumed  by  the  government,  such  as  integration  risk,  may  play  a 
large  part,  particularly  in  dramatic  program  cost  increases.  It  is  likely  that  both  possible 
causes  of  the  kurtosis  are  partially  responsible  for  the  non-normality  present  in  the  data. 
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Acquisition  Management 


Acquiring  Combat  Capability  via  Public-Private  Partnerships  (PPPs) 

BCA:  Contractor  vs.  Organic  Growth 

Defense  Industry  Consolidation 

EU-US  Defense  Industrial  Relationships 

Knowledge  Value  Added  (KVA)  +  Real  Options  (RO)  Applied  to 
Shipyard  Planning  Processes 

Managing  the  Services  Supply  Chain 

MOSA  Contracting  Implications 

Portfolio  Optimization  via  KVA  +  RO 

Private  Military  Sector 

Software  Requirements  for  OA 

Spiral  Development 

Strategy  for  Defense  Acquisition  Research 

The  Software,  Hardware  Asset  Reuse  Enterprise  (SHARE)  repository 


Contract  Management 


Commodity  Sourcing  Strategies 
Contracting  Government  Procurement  Functions 
Contractors  in  21st-century  Combat  Zone 
Joint  Contingency  Contracting 

Model  for  Optimizing  Contingency  Contracting,  Planning  and  Execution 

Navy  Contract  Writing  Guide 

Past  Performance  in  Source  Selection 

Strategic  Contingency  Contracting 

Transforming  DoD  Contract  Closeout 

USAF  Energy  Savings  Performance  Contracts 

USAF  IT  Commodity  Council 

USMC  Contingency  Contracting 


Financial  Management 
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■  Acquisitions  via  Leasing:  MPS  case 

■  Budget  Scoring 

■  Budgeting  for  Capabilities-based  Planning 

■  Capital  Budgeting  for  the  DoD 

■  Energy  Saving  Contracts/DoD  Mobile  Assets 

■  Financing  DoD  Budget  via  PPPs 

■  Lessons  from  Private  Sector  Capital  Budgeting  for  DoD  Acquisition 
Budgeting  Reform 

■  PPPs  and  Government  Financing 

■  ROI  of  Information  Warfare  Systems 

■  Special  Termination  Liability  in  MDAPs 

■  Strategic  Sourcing 

■  Transaction  Cost  Economics  (TCE)  to  Improve  Cost  Estimates 

Human  Resources 

■  Indefinite  Reenlistment 

■  Individual  Augmentation 

■  Learning  Management  Systems 

■  Moral  Conduct  Waivers  and  First-term  Attrition 

■  Retention 

■  The  Navy’s  Selective  Reenlistment  Bonus  (SRB)  Management  System 

■  Tuition  Assistance 

Logistics  Management 

■  Analysis  of  LAV  Depot  Maintenance 

■  Army  LOG  MOD 

■  ASDS  Product  Support  Analysis 
Cold-chain  Logistics 

Contractors  Supporting  Military  Operations 
DiffusionA/ariability  on  Vendor  Performance  Evaluation 
Evolutionary  Acquisition 

Lean  Six  Sigma  to  Reduce  Costs  and  Improve  Readiness 
Naval  Aviation  Maintenance  and  Process  Improvement  (2) 

Optimizing  CIWS  Lifecycle  Support  (LCS) 
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■  Outsourcing  the  Pearl  Harbor  MK-48  Intermediate  Maintenance 
Activity 

■  Pallet  Management  System 

■  PBL  (4) 

■  Privatization-NOSL/NAWCI 

■  RFID  (6) 

■  Risk  Analysis  for  Performance-based  Logistics 

■  R-TOC  AEGIS  Microwave  Power  Tubes 

■  Sense-and-Respond  Logistics  Network 

■  Strategic  Sourcing 

Program  Management 

■  Building  Collaborative  Capacity 

■  Business  Process  Reengineering  (BPR)  for  LCS  Mission  Module 
Acquisition 

■  Collaborative  IT  Tools  Leveraging  Competence 

■  Contractor  vs.  Organic  Support 

■  Knowledge,  Responsibilities  and  Decision  Rights  in  MDAPs 

■  KVA  Applied  to  AEGIS  and  SSDS 

■  Managing  the  Service  Supply  Chain 

■  Measuring  Uncertainty  in  Earned  Value 

■  Organizational  Modeling  and  Simulation 

■  Public-Private  Partnership 

■  Terminating  Your  Own  Program 

■  Utilizing  Collaborative  and  Three-dimensional  Imaging  Technology 

A  complete  listing  and  electronic  copies  of  published  research  are  available  on  our 
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